Oct 30 18:31:05.113046 ip-10-0-133-56 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 18:31:05.113059 ip-10-0-133-56 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 18:31:05.113066 ip-10-0-133-56 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 18:31:05.113307 ip-10-0-133-56 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 18:31:15.179221 ip-10-0-133-56 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 18:31:15.179237 ip-10-0-133-56 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 3a6723dcd77340978f1cbf2e00739c61 -- Oct 30 18:33:18.024871 ip-10-0-133-56 systemd[1]: Starting Kubernetes Kubelet... Oct 30 18:33:18.431263 ip-10-0-133-56 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:33:18.438588 ip-10-0-133-56 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 18:33:18.438588 ip-10-0-133-56 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:33:18.438588 ip-10-0-133-56 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:33:18.442080 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.441972 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:33:18.442080 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442079 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:33:18.442080 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442083 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:33:18.442184 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442087 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:33:18.442184 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442090 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:33:18.442184 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442093 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:33:18.442184 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442096 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:33:18.442184 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442098 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:33:18.442184 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442103 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:33:18.442184 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442108 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:33:18.442184 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442111 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:33:18.442184 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442115 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:33:18.442184 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442118 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:33:18.442184 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442121 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:33:18.442184 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442124 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:33:18.442184 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442127 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:33:18.442184 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442134 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:33:18.442184 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442137 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:33:18.442184 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442142 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:33:18.442184 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442146 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:33:18.442184 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442149 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:33:18.442184 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442151 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:33:18.442658 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442154 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:33:18.442658 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442157 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:33:18.442658 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442160 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:33:18.442658 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442162 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:33:18.442658 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442165 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:33:18.442658 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442167 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:33:18.442658 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442170 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:33:18.442658 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442173 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:33:18.442658 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442176 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:33:18.442658 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442179 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:33:18.442658 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442181 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:33:18.442658 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442184 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:33:18.442658 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442187 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:33:18.442658 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442190 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:33:18.442658 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442193 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:33:18.442658 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442197 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:33:18.442658 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442200 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:33:18.442658 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442202 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:33:18.442658 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442205 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:33:18.442658 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442208 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:33:18.443145 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442210 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:33:18.443145 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442213 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:33:18.443145 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442216 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:33:18.443145 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442219 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:33:18.443145 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442221 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:33:18.443145 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442224 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:33:18.443145 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442227 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:33:18.443145 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442229 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:33:18.443145 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442232 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:33:18.443145 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442234 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:33:18.443145 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442237 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:33:18.443145 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442239 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:33:18.443145 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442243 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:33:18.443145 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442245 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:33:18.443145 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442249 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:33:18.443145 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442252 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:33:18.443145 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442254 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:33:18.443145 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442257 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:33:18.443145 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442261 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:33:18.443145 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442264 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:33:18.443747 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442266 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:33:18.443747 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442269 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:33:18.443747 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442271 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:33:18.443747 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442274 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:33:18.443747 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442277 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:33:18.443747 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442280 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:33:18.443747 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442283 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:33:18.443747 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442285 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:33:18.443747 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442288 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:33:18.443747 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442291 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:33:18.443747 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442294 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:33:18.443747 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442297 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:33:18.443747 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442299 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:33:18.443747 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442303 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:33:18.443747 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442308 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:33:18.443747 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442311 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:33:18.443747 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442313 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:33:18.443747 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442316 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:33:18.443747 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442319 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:33:18.444211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442321 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:33:18.444211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442324 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:33:18.444211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442327 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:33:18.444211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442329 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:33:18.444211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442331 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:33:18.444211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442334 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:33:18.444211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442337 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:33:18.444211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442339 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:33:18.444211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442342 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:33:18.444211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442344 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:33:18.444211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442347 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:33:18.444211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442350 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:33:18.444211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442353 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:33:18.444211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442357 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:33:18.444211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442360 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:33:18.444211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442365 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:33:18.444211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442369 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:33:18.444211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442373 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:33:18.444211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442377 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:33:18.444734 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442380 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:33:18.444734 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442383 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:33:18.444734 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442385 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:33:18.444734 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442388 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:33:18.444734 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442391 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:33:18.444734 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.442394 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:33:18.444734 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443296 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:33:18.444734 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443306 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:33:18.444734 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443309 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:33:18.444734 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443324 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:33:18.444734 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443327 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:33:18.444734 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443330 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:33:18.444734 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443332 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:33:18.444734 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443335 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:33:18.444734 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443338 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:33:18.444734 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443341 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:33:18.444734 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443343 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:33:18.444734 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443346 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:33:18.444734 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443349 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:33:18.444734 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443352 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:33:18.445245 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443357 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:33:18.445245 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443360 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:33:18.445245 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443363 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:33:18.445245 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443365 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:33:18.445245 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443368 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:33:18.445245 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443370 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:33:18.445245 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443373 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:33:18.445245 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443375 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:33:18.445245 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443378 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:33:18.445245 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443380 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:33:18.445245 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443383 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:33:18.445245 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443385 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:33:18.445245 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443388 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:33:18.445245 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443391 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:33:18.445245 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443393 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:33:18.445245 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443396 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:33:18.445245 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443399 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:33:18.445245 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443401 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:33:18.445245 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443404 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:33:18.445245 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443407 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:33:18.445742 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443409 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:33:18.445742 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443412 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:33:18.445742 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443415 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:33:18.445742 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443420 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:33:18.445742 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443424 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:33:18.445742 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443427 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:33:18.445742 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443430 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:33:18.445742 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443432 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:33:18.445742 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443435 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:33:18.445742 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443438 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:33:18.445742 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443440 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:33:18.445742 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443442 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:33:18.445742 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443446 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:33:18.445742 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443449 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:33:18.445742 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443452 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:33:18.445742 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443454 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:33:18.445742 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443457 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:33:18.445742 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443460 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:33:18.445742 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443462 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:33:18.446211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443465 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:33:18.446211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443467 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:33:18.446211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443470 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:33:18.446211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443472 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:33:18.446211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443475 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:33:18.446211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443478 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:33:18.446211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443480 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:33:18.446211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443484 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:33:18.446211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443506 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:33:18.446211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443510 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:33:18.446211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443513 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:33:18.446211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443516 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:33:18.446211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443518 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:33:18.446211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443521 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:33:18.446211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443524 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:33:18.446211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443527 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:33:18.446211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443532 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:33:18.446211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443535 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:33:18.446211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443537 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:33:18.446211 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443540 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:33:18.446718 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443543 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:33:18.446718 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443546 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:33:18.446718 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443548 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:33:18.446718 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443551 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:33:18.446718 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443553 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:33:18.446718 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443558 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:33:18.446718 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443561 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:33:18.446718 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443563 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:33:18.446718 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443566 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:33:18.446718 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443568 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:33:18.446718 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443571 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:33:18.446718 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443573 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:33:18.446718 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443576 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:33:18.446718 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443579 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:33:18.446718 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443581 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:33:18.446718 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443584 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:33:18.446718 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443586 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:33:18.446718 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443589 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:33:18.446718 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443591 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:33:18.446718 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443595 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:33:18.447216 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443597 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:33:18.447216 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443600 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:33:18.447216 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443602 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:33:18.447216 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443605 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:33:18.447216 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443607 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:33:18.447216 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443610 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:33:18.447216 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443613 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:33:18.447216 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443616 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:33:18.447216 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443618 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:33:18.447216 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443622 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:33:18.447216 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443626 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:33:18.447216 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443629 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:33:18.447216 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.443633 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:33:18.447216 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444894 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:33:18.447216 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444903 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:33:18.447216 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444907 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:33:18.447216 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444911 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:33:18.447216 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444915 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:33:18.447216 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444918 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:33:18.447690 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444921 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:33:18.447690 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444923 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:33:18.447690 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444926 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:33:18.447690 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444929 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:33:18.447690 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444932 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:33:18.447690 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444934 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:33:18.447690 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444937 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:33:18.447690 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444939 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:33:18.447690 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444942 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:33:18.447690 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444945 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:33:18.447690 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444947 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:33:18.447690 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444949 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:33:18.447690 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444952 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:33:18.447690 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444955 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:33:18.447690 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444957 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:33:18.447690 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444960 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:33:18.447690 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444962 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:33:18.447690 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444965 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:33:18.447690 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444967 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:33:18.448149 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444970 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:33:18.448149 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444972 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:33:18.448149 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444975 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:33:18.448149 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444978 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:33:18.448149 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444981 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:33:18.448149 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444984 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:33:18.448149 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444986 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:33:18.448149 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444993 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:33:18.448149 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444996 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:33:18.448149 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.444998 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:33:18.448149 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445001 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:33:18.448149 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445004 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:33:18.448149 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445006 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:33:18.448149 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445010 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:33:18.448149 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445013 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:33:18.448149 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445016 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:33:18.448149 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445019 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:33:18.448149 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445022 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:33:18.448149 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445024 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:33:18.448659 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445027 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:33:18.448659 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445030 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:33:18.448659 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445032 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:33:18.448659 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445035 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:33:18.448659 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445037 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:33:18.448659 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445040 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:33:18.448659 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445043 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:33:18.448659 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445046 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:33:18.448659 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445048 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:33:18.448659 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445051 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:33:18.448659 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445053 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:33:18.448659 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445056 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:33:18.448659 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445058 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:33:18.448659 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445061 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:33:18.448659 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445063 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:33:18.448659 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445066 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:33:18.448659 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445069 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:33:18.448659 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445072 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:33:18.448659 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445076 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:33:18.448659 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445078 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:33:18.449155 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445081 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:33:18.449155 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445084 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:33:18.449155 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445086 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:33:18.449155 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445089 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:33:18.449155 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445091 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:33:18.449155 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445094 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:33:18.449155 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445096 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:33:18.449155 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445099 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:33:18.449155 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445101 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:33:18.449155 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445104 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:33:18.449155 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445108 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:33:18.449155 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445111 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:33:18.449155 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445114 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:33:18.449155 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445117 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:33:18.449155 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445120 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:33:18.449155 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445122 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:33:18.449155 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445125 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:33:18.449155 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445127 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:33:18.449155 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445130 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:33:18.449663 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445133 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:33:18.449663 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445135 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:33:18.449663 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445138 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:33:18.449663 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445140 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:33:18.449663 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445143 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:33:18.449663 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445146 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:33:18.449663 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445148 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:33:18.449663 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445150 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:33:18.449663 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445153 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:33:18.449663 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445156 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:33:18.449663 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445158 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:33:18.449663 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445163 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:33:18.449663 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445166 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:33:18.449663 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445169 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:33:18.449663 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445171 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:33:18.449663 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445174 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:33:18.449663 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445176 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:33:18.449663 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445178 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:33:18.449663 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445181 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:33:18.449663 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445183 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:33:18.450140 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445186 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:33:18.450140 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445188 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:33:18.450140 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.445191 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:33:18.450140 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446145 2571 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 18:33:18.450140 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446159 2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 18:33:18.450140 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446170 2571 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 18:33:18.450140 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446175 2571 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 18:33:18.450140 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446180 2571 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 18:33:18.450140 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446183 2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 18:33:18.450140 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446188 2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 18:33:18.450140 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446193 2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 18:33:18.450140 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446197 2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 18:33:18.450140 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446200 2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 18:33:18.450140 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446204 2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 18:33:18.450140 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446208 2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 18:33:18.450140 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446211 2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 18:33:18.450140 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446214 2571 flags.go:64] FLAG: --cgroup-root="" Oct 30 18:33:18.450140 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446217 2571 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 18:33:18.450140 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446221 2571 flags.go:64] FLAG: --client-ca-file="" Oct 30 18:33:18.450140 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446224 2571 flags.go:64] FLAG: --cloud-provider="external" Oct 30 18:33:18.450140 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446227 2571 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 18:33:18.450140 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446236 2571 flags.go:64] FLAG: --cluster-domain="" Oct 30 18:33:18.450140 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446240 2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 18:33:18.450140 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446243 2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 18:33:18.450721 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446247 2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 18:33:18.450721 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446251 2571 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 18:33:18.450721 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446255 2571 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 18:33:18.450721 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446258 2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 18:33:18.450721 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446261 2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 18:33:18.450721 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446265 2571 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 18:33:18.450721 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446268 2571 flags.go:64] FLAG: --contention-profiling="false" Oct 30 18:33:18.450721 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446271 2571 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 18:33:18.450721 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446274 2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 18:33:18.450721 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446277 2571 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 18:33:18.450721 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446280 2571 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 18:33:18.450721 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446285 2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 18:33:18.450721 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446288 2571 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 18:33:18.450721 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446292 2571 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 18:33:18.450721 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446295 2571 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 18:33:18.450721 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446298 2571 flags.go:64] FLAG: --enable-server="true" Oct 30 18:33:18.450721 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446301 2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 18:33:18.450721 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446305 2571 flags.go:64] FLAG: --event-burst="100" Oct 30 18:33:18.450721 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446309 2571 flags.go:64] FLAG: --event-qps="50" Oct 30 18:33:18.450721 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446312 2571 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 18:33:18.450721 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446316 2571 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 18:33:18.450721 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446319 2571 flags.go:64] FLAG: --eviction-hard="" Oct 30 18:33:18.450721 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446325 2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 18:33:18.450721 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446328 2571 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 18:33:18.451319 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446331 2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 18:33:18.451319 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446334 2571 flags.go:64] FLAG: --eviction-soft="" Oct 30 18:33:18.451319 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446337 2571 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 18:33:18.451319 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446341 2571 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 18:33:18.451319 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446344 2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 18:33:18.451319 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446349 2571 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 18:33:18.451319 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446352 2571 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 18:33:18.451319 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446355 2571 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 18:33:18.451319 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446358 2571 flags.go:64] FLAG: --feature-gates="" Oct 30 18:33:18.451319 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446364 2571 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 18:33:18.451319 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446367 2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 18:33:18.451319 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446370 2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 18:33:18.451319 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446373 2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 18:33:18.451319 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446376 2571 flags.go:64] FLAG: --healthz-port="10248" Oct 30 18:33:18.451319 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446379 2571 flags.go:64] FLAG: --help="false" Oct 30 18:33:18.451319 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446383 2571 flags.go:64] FLAG: --hostname-override="ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:33:18.451319 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446386 2571 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 18:33:18.451319 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446389 2571 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 18:33:18.451319 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446392 2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 18:33:18.451319 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446395 2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 18:33:18.451319 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446399 2571 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 18:33:18.451319 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446402 2571 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 18:33:18.451319 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446404 2571 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 18:33:18.451908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446407 2571 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 18:33:18.451908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446410 2571 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 18:33:18.451908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446413 2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 18:33:18.451908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446416 2571 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 18:33:18.451908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446419 2571 flags.go:64] FLAG: --kube-reserved="" Oct 30 18:33:18.451908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446422 2571 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 18:33:18.451908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446425 2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 18:33:18.451908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446428 2571 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 18:33:18.451908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446431 2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 18:33:18.451908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446434 2571 flags.go:64] FLAG: --lock-file="" Oct 30 18:33:18.451908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446437 2571 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 18:33:18.451908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446440 2571 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 18:33:18.451908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446443 2571 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 18:33:18.451908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446449 2571 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 18:33:18.451908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446453 2571 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 18:33:18.451908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446457 2571 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 18:33:18.451908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446459 2571 flags.go:64] FLAG: --logging-format="text" Oct 30 18:33:18.451908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446463 2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 18:33:18.451908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446466 2571 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 18:33:18.451908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446472 2571 flags.go:64] FLAG: --manifest-url="" Oct 30 18:33:18.451908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446475 2571 flags.go:64] FLAG: --manifest-url-header="" Oct 30 18:33:18.451908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446481 2571 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 18:33:18.451908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446485 2571 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 18:33:18.451908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446506 2571 flags.go:64] FLAG: --max-pods="110" Oct 30 18:33:18.451908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446509 2571 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 18:33:18.452545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446513 2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 18:33:18.452545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446516 2571 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 18:33:18.452545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446520 2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 18:33:18.452545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446523 2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 18:33:18.452545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446526 2571 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 18:33:18.452545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446528 2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 18:33:18.452545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446538 2571 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 18:33:18.452545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446542 2571 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 18:33:18.452545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446545 2571 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 18:33:18.452545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446548 2571 flags.go:64] FLAG: --pod-cidr="" Oct 30 18:33:18.452545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446551 2571 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 18:33:18.452545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446554 2571 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 18:33:18.452545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446557 2571 flags.go:64] FLAG: --pods-per-core="0" Oct 30 18:33:18.452545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446559 2571 flags.go:64] FLAG: --port="10250" Oct 30 18:33:18.452545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446563 2571 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 18:33:18.452545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446565 2571 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0badc0c146ab7f8f0" Oct 30 18:33:18.452545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446569 2571 flags.go:64] FLAG: --qos-reserved="" Oct 30 18:33:18.452545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446572 2571 flags.go:64] FLAG: --read-only-port="10255" Oct 30 18:33:18.452545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446575 2571 flags.go:64] FLAG: --register-node="true" Oct 30 18:33:18.452545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446578 2571 flags.go:64] FLAG: --register-with-taints="" Oct 30 18:33:18.452545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446582 2571 flags.go:64] FLAG: --registry-burst="10" Oct 30 18:33:18.452545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446586 2571 flags.go:64] FLAG: --registry-qps="5" Oct 30 18:33:18.452545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446589 2571 flags.go:64] FLAG: --reserved-cpus="" Oct 30 18:33:18.452545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446592 2571 flags.go:64] FLAG: --reserved-memory="" Oct 30 18:33:18.452545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446596 2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 18:33:18.452545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446600 2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 18:33:18.453270 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446604 2571 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 18:33:18.453270 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446608 2571 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 18:33:18.453270 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446611 2571 flags.go:64] FLAG: --runonce="false" Oct 30 18:33:18.453270 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446614 2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 18:33:18.453270 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446617 2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 18:33:18.453270 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446620 2571 flags.go:64] FLAG: --seccomp-default="false" Oct 30 18:33:18.453270 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446623 2571 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 18:33:18.453270 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446626 2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 18:33:18.453270 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446629 2571 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 18:33:18.453270 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446632 2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 18:33:18.453270 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446636 2571 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 18:33:18.453270 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446639 2571 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 18:33:18.453270 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446641 2571 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 18:33:18.453270 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446644 2571 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 18:33:18.453270 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446647 2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 18:33:18.453270 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446650 2571 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 18:33:18.453270 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446653 2571 flags.go:64] FLAG: --system-cgroups="" Oct 30 18:33:18.453270 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446656 2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 18:33:18.453270 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446661 2571 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 18:33:18.453270 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446664 2571 flags.go:64] FLAG: --tls-cert-file="" Oct 30 18:33:18.453270 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446667 2571 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 18:33:18.453270 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446672 2571 flags.go:64] FLAG: --tls-min-version="" Oct 30 18:33:18.453270 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446674 2571 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 18:33:18.453270 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446677 2571 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 18:33:18.453270 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446680 2571 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 18:33:18.453905 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446683 2571 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 18:33:18.453905 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446685 2571 flags.go:64] FLAG: --v="2" Oct 30 18:33:18.453905 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446694 2571 flags.go:64] FLAG: --version="false" Oct 30 18:33:18.453905 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446699 2571 flags.go:64] FLAG: --vmodule="" Oct 30 18:33:18.453905 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446703 2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 18:33:18.453905 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.446706 2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 18:33:18.453905 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446869 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:33:18.453905 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446875 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:33:18.453905 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446878 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:33:18.453905 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446882 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:33:18.453905 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446885 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:33:18.453905 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446887 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:33:18.453905 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446890 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:33:18.453905 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446892 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:33:18.453905 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446895 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:33:18.453905 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446898 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:33:18.453905 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446900 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:33:18.453905 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446903 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:33:18.453905 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446905 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:33:18.453905 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446907 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:33:18.453905 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446910 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:33:18.454409 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446913 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:33:18.454409 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446916 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:33:18.454409 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446918 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:33:18.454409 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446921 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:33:18.454409 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446924 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:33:18.454409 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446926 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:33:18.454409 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446929 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:33:18.454409 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446932 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:33:18.454409 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446936 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:33:18.454409 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446940 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:33:18.454409 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446944 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:33:18.454409 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446946 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:33:18.454409 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446949 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:33:18.454409 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446953 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:33:18.454409 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446956 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:33:18.454409 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446958 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:33:18.454409 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446961 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:33:18.454409 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446963 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:33:18.454409 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446967 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:33:18.454409 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446970 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:33:18.454925 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446973 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:33:18.454925 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446976 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:33:18.454925 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446978 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:33:18.454925 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446981 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:33:18.454925 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446984 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:33:18.454925 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446986 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:33:18.454925 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446989 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:33:18.454925 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446992 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:33:18.454925 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446994 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:33:18.454925 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446997 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:33:18.454925 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.446999 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:33:18.454925 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447002 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:33:18.454925 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447005 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:33:18.454925 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447009 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:33:18.454925 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447012 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:33:18.454925 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447015 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:33:18.454925 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447018 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:33:18.454925 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447021 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:33:18.454925 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447023 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:33:18.455384 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447026 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:33:18.455384 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447029 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:33:18.455384 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447031 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:33:18.455384 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447034 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:33:18.455384 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447036 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:33:18.455384 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447038 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:33:18.455384 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447042 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:33:18.455384 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447045 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:33:18.455384 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447048 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:33:18.455384 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447051 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:33:18.455384 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447053 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:33:18.455384 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447057 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:33:18.455384 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447060 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:33:18.455384 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447063 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:33:18.455384 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447065 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:33:18.455384 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447069 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:33:18.455384 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447071 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:33:18.455384 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447074 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:33:18.455384 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447077 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:33:18.455384 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447079 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:33:18.455910 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447082 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:33:18.455910 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447084 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:33:18.455910 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447087 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:33:18.455910 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447089 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:33:18.455910 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447092 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:33:18.455910 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447094 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:33:18.455910 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447097 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:33:18.455910 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447100 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:33:18.455910 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447103 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:33:18.455910 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447106 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:33:18.455910 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447109 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:33:18.455910 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447111 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:33:18.455910 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447114 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:33:18.455910 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447116 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:33:18.455910 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447119 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:33:18.455910 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447121 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:33:18.455910 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447124 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:33:18.455910 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447126 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:33:18.455910 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447131 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:33:18.455910 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447134 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:33:18.456378 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447137 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:33:18.456378 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447139 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:33:18.456378 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447142 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:33:18.456378 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447145 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:33:18.456378 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447149 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:33:18.456378 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447151 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:33:18.456378 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447153 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:33:18.456378 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447156 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:33:18.456378 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447159 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:33:18.456378 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447162 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:33:18.456378 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447164 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:33:18.456378 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447167 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:33:18.456378 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.447198 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:33:18.456378 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447236 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:33:18.456378 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447242 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:33:18.456378 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447245 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:33:18.456813 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447247 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:33:18.456813 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447250 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:33:18.456813 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447253 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:33:18.456813 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447255 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:33:18.456813 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447258 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:33:18.456813 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447261 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:33:18.456813 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447263 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:33:18.456813 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447266 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:33:18.456813 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447268 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:33:18.456813 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447271 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:33:18.456813 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447273 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:33:18.456813 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447276 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:33:18.456813 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447278 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:33:18.456813 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447281 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:33:18.456813 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447283 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:33:18.456813 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447287 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:33:18.456813 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447289 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:33:18.456813 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447292 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:33:18.456813 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447294 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:33:18.456813 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447298 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:33:18.457299 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447301 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:33:18.457299 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447304 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:33:18.457299 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447306 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:33:18.457299 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447309 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:33:18.457299 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447311 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:33:18.457299 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447314 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:33:18.457299 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447317 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:33:18.457299 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447319 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:33:18.457299 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447321 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:33:18.457299 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447324 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:33:18.457299 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447326 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:33:18.457299 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447329 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:33:18.457299 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447331 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:33:18.457299 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447334 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:33:18.457299 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447336 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:33:18.457299 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447339 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:33:18.457299 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447341 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:33:18.457299 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447343 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:33:18.457299 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447346 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:33:18.457785 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447348 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:33:18.457785 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447351 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:33:18.457785 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447353 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:33:18.457785 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447356 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:33:18.457785 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447359 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:33:18.457785 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447362 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:33:18.457785 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447366 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:33:18.457785 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447369 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:33:18.457785 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447372 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:33:18.457785 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447375 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:33:18.457785 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447378 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:33:18.457785 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447380 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:33:18.457785 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447384 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:33:18.457785 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447387 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:33:18.457785 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447389 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:33:18.457785 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447392 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:33:18.457785 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447394 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:33:18.457785 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447396 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:33:18.457785 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447399 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:33:18.457785 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447402 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:33:18.458274 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447405 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:33:18.458274 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447407 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:33:18.458274 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447409 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:33:18.458274 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447412 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:33:18.458274 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447414 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:33:18.458274 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447417 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:33:18.458274 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447419 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:33:18.458274 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447422 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:33:18.458274 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447424 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:33:18.458274 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447427 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:33:18.458274 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447431 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:33:18.458274 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447434 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:33:18.458274 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447437 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:33:18.458274 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447439 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:33:18.458274 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447443 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:33:18.458274 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447445 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:33:18.458274 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447448 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:33:18.458274 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447451 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:33:18.458274 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447453 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:33:18.458814 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447456 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:33:18.458814 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447458 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:33:18.458814 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447461 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:33:18.458814 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447463 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:33:18.458814 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447466 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:33:18.458814 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447471 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:33:18.458814 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447474 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:33:18.458814 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447476 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:33:18.458814 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447479 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:33:18.458814 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447482 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:33:18.458814 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447484 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:33:18.458814 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447504 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:33:18.458814 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447508 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:33:18.458814 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447511 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:33:18.458814 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447514 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:33:18.458814 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447517 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:33:18.458814 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447519 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:33:18.458814 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447522 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:33:18.458814 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447525 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:33:18.458814 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447527 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:33:18.459295 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447531 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:33:18.459295 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447533 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:33:18.459295 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447536 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:33:18.459295 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447538 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:33:18.459295 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.447541 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:33:18.459295 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.454871 2571 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 18:33:18.459295 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.454897 2571 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 18:33:18.459295 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.455653 2571 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 18:33:18.459295 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.455662 2571 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 18:33:18.459295 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455734 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:33:18.459295 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455741 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:33:18.459295 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455745 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:33:18.459295 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455747 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:33:18.459295 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455750 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:33:18.459295 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455753 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:33:18.459295 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455756 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:33:18.459295 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455759 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:33:18.459295 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455764 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:33:18.459295 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455768 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:33:18.459800 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455771 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:33:18.459800 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455774 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:33:18.459800 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455777 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:33:18.459800 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455780 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:33:18.459800 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455783 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:33:18.459800 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455786 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:33:18.459800 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455789 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:33:18.459800 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455802 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:33:18.459800 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455805 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:33:18.459800 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455808 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:33:18.459800 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455810 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:33:18.459800 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455813 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:33:18.459800 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455816 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:33:18.459800 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455820 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:33:18.459800 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455824 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:33:18.459800 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455827 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:33:18.459800 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455829 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:33:18.459800 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455832 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:33:18.459800 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455836 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:33:18.460269 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455843 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:33:18.460269 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455846 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:33:18.460269 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455849 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:33:18.460269 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455852 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:33:18.460269 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455854 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:33:18.460269 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455857 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:33:18.460269 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455859 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:33:18.460269 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455862 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:33:18.460269 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455865 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:33:18.460269 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455868 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:33:18.460269 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455870 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:33:18.460269 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455873 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:33:18.460269 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455875 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:33:18.460269 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455878 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:33:18.460269 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455880 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:33:18.460269 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455883 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:33:18.460269 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455885 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:33:18.460269 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455889 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:33:18.460269 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455891 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:33:18.460269 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455894 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:33:18.460811 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455896 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:33:18.460811 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455899 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:33:18.460811 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455901 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:33:18.460811 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455904 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:33:18.460811 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455907 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:33:18.460811 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455910 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:33:18.460811 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455913 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:33:18.460811 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455916 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:33:18.460811 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455919 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:33:18.460811 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455921 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:33:18.460811 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455924 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:33:18.460811 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455927 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:33:18.460811 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455929 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:33:18.460811 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455932 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:33:18.460811 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455935 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:33:18.460811 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455938 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:33:18.460811 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455941 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:33:18.460811 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455943 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:33:18.460811 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455946 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:33:18.460811 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455948 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:33:18.461314 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455951 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:33:18.461314 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455953 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:33:18.461314 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455955 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:33:18.461314 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455958 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:33:18.461314 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455961 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:33:18.461314 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455963 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:33:18.461314 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455965 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:33:18.461314 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455968 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:33:18.461314 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455971 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:33:18.461314 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455974 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:33:18.461314 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455976 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:33:18.461314 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455979 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:33:18.461314 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455981 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:33:18.461314 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455984 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:33:18.461314 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455986 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:33:18.461314 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455989 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:33:18.461314 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455992 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:33:18.461314 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455994 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:33:18.461314 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455997 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:33:18.461314 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.455999 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:33:18.461816 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456002 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:33:18.461816 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456005 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:33:18.461816 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456007 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:33:18.461816 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456010 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:33:18.461816 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456013 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:33:18.461816 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456015 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:33:18.461816 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456019 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:33:18.461816 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456021 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:33:18.461816 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456024 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:33:18.461816 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456026 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:33:18.461816 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456029 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:33:18.461816 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456031 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:33:18.461816 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456033 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:33:18.461816 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456036 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:33:18.461816 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456038 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:33:18.461816 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456041 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:33:18.461816 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456043 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:33:18.462239 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.456084 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:33:18.462239 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456237 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:33:18.462239 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456242 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:33:18.462239 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456246 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:33:18.462239 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456248 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:33:18.462239 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456252 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:33:18.462239 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456255 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:33:18.462239 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456257 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:33:18.462239 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456260 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:33:18.462239 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456262 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:33:18.462239 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456265 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:33:18.462239 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456267 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:33:18.462239 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456270 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:33:18.462239 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456272 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:33:18.462239 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456275 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:33:18.462239 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456277 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:33:18.462681 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456279 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:33:18.462681 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456282 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:33:18.462681 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456286 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:33:18.462681 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456288 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:33:18.462681 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456291 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:33:18.462681 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456293 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:33:18.462681 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456296 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:33:18.462681 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456299 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:33:18.462681 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456302 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:33:18.462681 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456304 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:33:18.462681 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456307 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:33:18.462681 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456309 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:33:18.462681 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456312 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:33:18.462681 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456315 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:33:18.462681 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456317 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:33:18.462681 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456320 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:33:18.462681 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456322 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:33:18.462681 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456325 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:33:18.462681 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456328 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:33:18.462681 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456330 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:33:18.463180 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456333 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:33:18.463180 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456336 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:33:18.463180 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456339 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:33:18.463180 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456341 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:33:18.463180 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456344 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:33:18.463180 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456347 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:33:18.463180 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456350 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:33:18.463180 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456352 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:33:18.463180 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456355 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:33:18.463180 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456358 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:33:18.463180 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456360 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:33:18.463180 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456362 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:33:18.463180 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456364 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:33:18.463180 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456367 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:33:18.463180 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456370 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:33:18.463180 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456373 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:33:18.463180 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456375 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:33:18.463180 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456378 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:33:18.463180 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456381 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:33:18.463660 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456384 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:33:18.463660 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456387 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:33:18.463660 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456389 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:33:18.463660 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456392 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:33:18.463660 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456395 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:33:18.463660 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456397 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:33:18.463660 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456401 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:33:18.463660 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456405 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:33:18.463660 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456408 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:33:18.463660 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456411 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:33:18.463660 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456414 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:33:18.463660 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456417 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:33:18.463660 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456419 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:33:18.463660 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456422 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:33:18.463660 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456426 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:33:18.463660 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456430 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:33:18.463660 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456434 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:33:18.463660 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456436 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:33:18.463660 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456439 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:33:18.464167 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456442 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:33:18.464167 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456444 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:33:18.464167 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456447 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:33:18.464167 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456450 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:33:18.464167 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456452 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:33:18.464167 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456454 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:33:18.464167 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456457 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:33:18.464167 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456460 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:33:18.464167 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456463 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:33:18.464167 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456465 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:33:18.464167 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456468 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:33:18.464167 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456470 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:33:18.464167 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456472 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:33:18.464167 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456475 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:33:18.464167 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456478 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:33:18.464167 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456481 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:33:18.464167 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456483 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:33:18.464167 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456485 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:33:18.464167 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456503 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:33:18.464929 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456506 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:33:18.464929 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456509 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:33:18.464929 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456512 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:33:18.464929 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456515 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:33:18.464929 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456518 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:33:18.464929 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456520 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:33:18.464929 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456523 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:33:18.464929 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456526 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:33:18.464929 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456529 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:33:18.464929 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456531 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:33:18.464929 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456534 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:33:18.464929 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456536 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:33:18.464929 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456539 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:33:18.464929 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456542 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:33:18.464929 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.456566 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:33:18.464929 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456599 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:33:18.465340 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456605 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:33:18.465340 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456609 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:33:18.465340 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456613 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:33:18.465340 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456616 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:33:18.465340 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456619 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:33:18.465340 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456622 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:33:18.465340 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456624 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:33:18.465340 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456628 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:33:18.465340 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456633 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:33:18.465340 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456635 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:33:18.465340 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456638 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:33:18.465340 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456641 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:33:18.465340 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456644 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:33:18.465340 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456646 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:33:18.465340 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456649 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:33:18.465340 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456652 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:33:18.465340 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456654 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:33:18.465340 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456657 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:33:18.465340 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456660 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:33:18.465992 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456662 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:33:18.465992 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456665 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:33:18.465992 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456668 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:33:18.465992 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456670 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:33:18.465992 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456673 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:33:18.465992 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456675 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:33:18.465992 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456678 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:33:18.465992 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456680 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:33:18.465992 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456683 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:33:18.465992 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456685 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:33:18.465992 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456687 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:33:18.465992 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456690 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:33:18.465992 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456692 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:33:18.465992 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456695 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:33:18.465992 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456698 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:33:18.465992 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456700 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:33:18.465992 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456703 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:33:18.465992 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456706 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:33:18.465992 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456709 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:33:18.465992 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456712 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:33:18.466460 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456714 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:33:18.466460 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456717 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:33:18.466460 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456719 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:33:18.466460 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456722 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:33:18.466460 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456724 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:33:18.466460 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456727 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:33:18.466460 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456730 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:33:18.466460 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456733 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:33:18.466460 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456735 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:33:18.466460 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456738 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:33:18.466460 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456746 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:33:18.466460 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456749 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:33:18.466460 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456751 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:33:18.466460 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456753 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:33:18.466460 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456757 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:33:18.466460 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456759 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:33:18.466460 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456762 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:33:18.466460 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456765 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:33:18.466460 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456767 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:33:18.466460 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456770 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:33:18.466958 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456773 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:33:18.466958 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456786 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:33:18.466958 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456789 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:33:18.466958 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456792 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:33:18.466958 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456795 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:33:18.466958 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456797 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:33:18.466958 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456800 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:33:18.466958 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456802 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:33:18.466958 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456805 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:33:18.466958 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456808 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:33:18.466958 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456811 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:33:18.466958 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456813 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:33:18.466958 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456816 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:33:18.466958 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456819 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:33:18.466958 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456821 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:33:18.466958 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456824 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:33:18.466958 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456827 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:33:18.466958 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456829 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:33:18.466958 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456832 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:33:18.466958 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456836 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:33:18.467471 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456839 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:33:18.467471 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456841 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:33:18.467471 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456845 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:33:18.467471 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456847 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:33:18.467471 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456850 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:33:18.467471 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456852 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:33:18.467471 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456855 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:33:18.467471 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456857 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:33:18.467471 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456860 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:33:18.467471 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456862 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:33:18.467471 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456865 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:33:18.467471 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456867 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:33:18.467471 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456870 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:33:18.467471 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456872 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:33:18.467471 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456874 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:33:18.467471 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456877 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:33:18.467471 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456879 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:33:18.467471 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456882 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:33:18.467471 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456884 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:33:18.468010 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456887 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:33:18.468010 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456891 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:33:18.468010 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456894 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:33:18.468010 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456896 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:33:18.468010 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456899 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:33:18.468010 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456901 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:33:18.468010 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:18.456904 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:33:18.468010 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.456973 2571 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 18:33:18.468010 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.459046 2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 18:33:18.468010 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.460284 2571 server.go:1014] "Starting client certificate rotation" Oct 30 18:33:18.468010 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.460392 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 18:33:18.468010 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.461216 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 18:33:18.468010 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.462540 2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:33:18.468988 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.468959 2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:33:18.479117 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.479090 2571 log.go:25] "Validated CRI v1 runtime API" Oct 30 18:33:18.482430 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.482399 2571 log.go:25] "Validated CRI v1 image API" Oct 30 18:33:18.483410 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.483377 2571 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 18:33:18.486362 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.486333 2571 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b1277cd2-d2fa-44bb-8a0d-3517cc2f9b60:/dev/nvme0n1p3 d18d8c20-68c9-4e92-98d0-3d37f1dbab67:/dev/nvme0n1p4] Oct 30 18:33:18.486362 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.486357 2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 18:33:18.492665 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.492517 2571 manager.go:217] Machine: {Timestamp:2026-10-30 18:33:18.49109511 +0000 UTC m=+0.343440919 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098588 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2770dd80d717336ce1649831bda449 SystemUUID:ec2770dd-80d7-1733-6ce1-649831bda449 BootID:3a6723dc-d773-4097-8f1c-bf2e00739c61 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2f:44:1d:f7:87 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2f:44:1d:f7:87 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:24:72:ba:9d:a2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 18:33:18.493150 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.493136 2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 18:33:18.493273 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.493252 2571 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 18:33:18.497253 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.497204 2571 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 18:33:18.497426 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.497255 2571 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-56.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 18:33:18.497471 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.497428 2571 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 18:33:18.497471 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.497438 2571 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 18:33:18.497471 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.497469 2571 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:33:18.498590 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.498575 2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:33:18.499665 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.499648 2571 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 18:33:18.502534 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.502515 2571 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 18:33:18.502707 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.502697 2571 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 18:33:18.503405 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.503395 2571 kubelet.go:493] "Attempting to sync node with API server" Oct 30 18:33:18.503438 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.503413 2571 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 18:33:18.503438 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.503429 2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 18:33:18.503534 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.503442 2571 kubelet.go:405] "Adding apiserver pod source" Oct 30 18:33:18.503534 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.503451 2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 18:33:18.503874 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.503860 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:33:18.506565 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.506544 2571 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 18:33:18.511861 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.511830 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:33:18.512654 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.512634 2571 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 18:33:18.512771 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.512680 2571 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 18:33:18.512923 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.512909 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 18:33:18.512987 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.512931 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 18:33:18.512987 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.512941 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 18:33:18.512987 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.512950 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 18:33:18.512987 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.512959 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 18:33:18.512987 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.512969 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 18:33:18.513694 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.513684 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 18:33:18.513760 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.513699 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 18:33:18.513760 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.513711 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 18:33:18.513760 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.513720 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 18:33:18.513760 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.513734 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 18:33:18.513760 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.513748 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 18:33:18.514308 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.514297 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 18:33:18.514363 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.514314 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 18:33:18.518658 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.518626 2571 server.go:1263] "Started kubelet" Oct 30 18:33:18.518789 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.518724 2571 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 18:33:18.518867 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.518755 2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 18:33:18.518867 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.518826 2571 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 18:33:18.519585 ip-10-0-133-56 systemd[1]: Started Kubernetes Kubelet. Oct 30 18:33:18.520156 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.519922 2571 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 18:33:18.521044 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.521020 2571 server.go:317] "Adding debug handlers to kubelet server" Oct 30 18:33:18.525366 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:18.525341 2571 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 18:33:18.525526 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.525508 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 18:33:18.526362 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.526341 2571 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 18:33:18.526541 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:18.526525 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-56.us-west-1.compute.internal\" not found" Oct 30 18:33:18.526835 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.526791 2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 18:33:18.526835 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.526795 2571 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 18:33:18.526835 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.526822 2571 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 18:33:18.526995 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.526878 2571 reconstruct.go:97] "Volume reconstruction finished" Oct 30 18:33:18.526995 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.526887 2571 reconciler.go:29] "Reconciler: start to sync state" Oct 30 18:33:18.527335 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.527322 2571 factory.go:55] Registering systemd factory Oct 30 18:33:18.527400 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.527337 2571 factory.go:223] Registration of the systemd container factory successfully Oct 30 18:33:18.527568 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.527557 2571 factory.go:153] Registering CRI-O factory Oct 30 18:33:18.527568 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.527569 2571 factory.go:223] Registration of the crio container factory successfully Oct 30 18:33:18.527637 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.527612 2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 18:33:18.527637 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.527633 2571 factory.go:103] Registering Raw factory Oct 30 18:33:18.527688 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.527643 2571 manager.go:1199] Started watching for new ooms in manager Oct 30 18:33:18.528531 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.528516 2571 manager.go:319] Starting recovery of all containers Oct 30 18:33:18.542180 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.542160 2571 manager.go:324] Recovery completed Oct 30 18:33:18.543590 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:18.543476 2571 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 30 18:33:18.546472 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.546458 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:33:18.548520 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.548468 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:33:18.548621 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.548542 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:33:18.548621 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.548561 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 18:33:18.549191 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.549176 2571 cpu_manager.go:226] "Starting" policy="none" Oct 30 18:33:18.549191 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.549187 2571 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 18:33:18.549283 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.549208 2571 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 18:33:18.549283 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.549229 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:33:18.550147 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.550131 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:33:18.550196 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.550162 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:33:18.550196 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.550173 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 18:33:18.550617 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.550602 2571 policy_none.go:50] "Start" Oct 30 18:33:18.550617 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.550619 2571 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 18:33:18.550749 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.550629 2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 18:33:18.552821 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.552805 2571 policy_none.go:44] "Start" Oct 30 18:33:18.598276 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.598209 2571 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 18:33:18.601040 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.601021 2571 manager.go:341] "Starting Device Plugin manager" Oct 30 18:33:18.601241 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:18.601060 2571 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 18:33:18.601294 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.601246 2571 server.go:87] "Starting device plugin registration server" Oct 30 18:33:18.601539 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.601526 2571 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 18:33:18.601615 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.601541 2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 18:33:18.601839 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.601644 2571 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 18:33:18.601839 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.601788 2571 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 18:33:18.601839 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.601799 2571 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 18:33:18.602950 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:18.602930 2571 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 18:33:18.603047 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:18.602980 2571 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-56.us-west-1.compute.internal\" not found" Oct 30 18:33:18.663571 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.663539 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-56.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 18:33:18.663571 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:18.663540 2571 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-56.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 18:33:18.664584 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.663987 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 18:33:18.664627 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:18.663539 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-56.us-west-1.compute.internal.18e362561724c72d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-56.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-56.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 18:33:18.518585133 +0000 UTC m=+0.370930941,LastTimestamp:2026-10-30 18:33:18.518585133 +0000 UTC m=+0.370930941,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-56.us-west-1.compute.internal,}" Oct 30 18:33:18.665516 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.665483 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 18:33:18.665561 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.665522 2571 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 18:33:18.665561 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.665542 2571 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 18:33:18.665630 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:18.665592 2571 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 18:33:18.701751 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.701674 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:33:18.702756 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.702734 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:33:18.702891 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.702776 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:33:18.702891 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.702792 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 18:33:18.702891 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.702830 2571 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:33:18.747414 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.747374 2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 18:33:18.747633 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.747438 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nhlrr" Oct 30 18:33:18.765734 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.765694 2571 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal"] Oct 30 18:33:18.765920 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.765785 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:33:18.766866 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.766848 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:33:18.766941 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.766881 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:33:18.766941 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.766891 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 18:33:18.768080 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.768065 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:33:18.768266 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.768249 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:33:18.768338 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.768294 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:33:18.768991 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.768976 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:33:18.768991 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.768989 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:33:18.769118 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.769004 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:33:18.769118 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.769011 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:33:18.769118 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.769014 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 18:33:18.769118 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.769037 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 18:33:18.770258 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.770240 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:33:18.770310 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.770289 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:33:18.771341 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.771322 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:33:18.771450 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.771358 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:33:18.771450 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.771372 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 18:33:18.791181 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:18.791143 2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-56.us-west-1.compute.internal\" not found" node="ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:33:18.795691 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:18.795663 2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-56.us-west-1.compute.internal\" not found" node="ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:33:18.825160 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.825127 2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:33:18.826578 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.826556 2571 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:33:18.829645 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.829608 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76ae57dd60561f1d7425076e8b67e7e3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal\" (UID: \"76ae57dd60561f1d7425076e8b67e7e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:33:18.829756 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.829653 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76ae57dd60561f1d7425076e8b67e7e3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal\" (UID: \"76ae57dd60561f1d7425076e8b67e7e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:33:18.829756 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.829674 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d156aeb8e5a3750f522b65c4725c3f82-config\") pod \"kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal\" (UID: \"d156aeb8e5a3750f522b65c4725c3f82\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:33:18.924302 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.924273 2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:33:18.926971 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.926938 2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:33:18.927283 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.927244 2571 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:33:18.927360 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:18.927293 2571 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-56.us-west-1.compute.internal\": node \"ip-10-0-133-56.us-west-1.compute.internal\" not found" Oct 30 18:33:18.929972 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.929940 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76ae57dd60561f1d7425076e8b67e7e3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal\" (UID: \"76ae57dd60561f1d7425076e8b67e7e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:33:18.929972 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.929971 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d156aeb8e5a3750f522b65c4725c3f82-config\") pod \"kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal\" (UID: \"d156aeb8e5a3750f522b65c4725c3f82\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:33:18.930125 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.929989 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76ae57dd60561f1d7425076e8b67e7e3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal\" (UID: \"76ae57dd60561f1d7425076e8b67e7e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:33:18.930125 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.930053 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76ae57dd60561f1d7425076e8b67e7e3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal\" (UID: \"76ae57dd60561f1d7425076e8b67e7e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:33:18.930125 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.930059 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d156aeb8e5a3750f522b65c4725c3f82-config\") pod \"kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal\" (UID: \"d156aeb8e5a3750f522b65c4725c3f82\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:33:18.930125 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:18.930057 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76ae57dd60561f1d7425076e8b67e7e3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal\" (UID: \"76ae57dd60561f1d7425076e8b67e7e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:33:19.026837 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.026724 2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:33:19.094254 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.094222 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:33:19.098919 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.098612 2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:33:19.098919 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.098724 2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:33:19.099670 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.099656 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:33:19.156908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.156882 2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:33:19.459781 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.459746 2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 18:33:19.460322 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.459920 2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 18:33:19.460322 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.459964 2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 18:33:19.460322 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.459969 2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 18:33:19.460322 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:19.460003 2571 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a4d303792d30e478da0cee90b09d474b-fa61d523e62daa65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-133-56.us-west-1.compute.internal.18e3625618ee4431\": read tcp 10.0.133.56:52910->77.114.47.6:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-133-56.us-west-1.compute.internal.18e3625618ee4431 default 4846 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-56.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-56.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-56.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 18:33:18 +0000 UTC,LastTimestamp:2026-10-30 18:33:18.702796786 +0000 UTC m=+0.555142596,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-56.us-west-1.compute.internal,}" Oct 30 18:33:19.503964 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.503934 2571 apiserver.go:52] "Watching apiserver" Oct 30 18:33:19.526230 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.526198 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 18:33:19.751619 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.751536 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:28:20 +0000 UTC" deadline="2028-08-04 14:37:01.278321246 +0000 UTC" Oct 30 18:33:19.751619 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.751569 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15452h3m41.52675558s" Oct 30 18:33:19.772914 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.772883 2571 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 18:33:19.773235 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.773215 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-ptpzt","kube-system/kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-996pn","openshift-image-registry/node-ca-7c9gt","openshift-multus/multus-f2745","openshift-network-diagnostics/network-check-target-hh4r5","openshift-network-operator/iptables-alerter-qnqxv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-flgwv","kube-system/global-pull-secret-syncer-frkss","kube-system/konnectivity-agent-sd88x","openshift-dns/node-resolver-7jq9m","openshift-multus/multus-additional-cni-plugins-nxpzg"] Oct 30 18:33:19.774763 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.774746 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.775886 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.775863 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.776821 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.776803 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7c9gt" Oct 30 18:33:19.777931 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.777914 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f2745" Oct 30 18:33:19.778972 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.778955 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:19.779048 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:19.779028 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hh4r5" podUID="17dae92a-ae2b-4c09-8001-48a81a8cf809" Oct 30 18:33:19.779887 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.779875 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qnqxv" Oct 30 18:33:19.780891 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.780875 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:33:19.782278 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.782264 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:19.782339 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:19.782324 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flgwv" podUID="c84bbebc-7b6d-4e22-aacd-c98586717109" Oct 30 18:33:19.783425 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.783410 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:19.783476 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:19.783463 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frkss" podUID="429722d3-91b1-4c92-b78e-19a12bc5bac9" Oct 30 18:33:19.784623 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.784609 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sd88x" Oct 30 18:33:19.786071 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.785915 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7jq9m" Oct 30 18:33:19.788441 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.788419 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:33:19.828388 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.828358 2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 18:33:19.829080 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.828949 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 18:33:19.829080 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.829045 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5qqcs\"" Oct 30 18:33:19.829247 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.829168 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jkxcr\"" Oct 30 18:33:19.830073 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.830050 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qb49q\"" Oct 30 18:33:19.830525 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.830508 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-b7gnv\"" Oct 30 18:33:19.831021 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.831007 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 18:33:19.831241 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.831171 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 18:33:19.831241 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.831210 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 18:33:19.831445 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.831244 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 18:33:19.831445 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.831253 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 18:33:19.831445 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.831258 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 18:33:19.831445 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.831330 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:33:19.831445 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.831336 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 18:33:19.831445 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.831363 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 18:33:19.831445 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.831397 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 18:33:19.831445 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.831429 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 18:33:19.831718 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.831457 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 18:33:19.831718 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.831564 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 18:33:19.831718 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.831650 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 18:33:19.832169 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.832155 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 18:33:19.835430 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.835409 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d258432-7e24-4b20-a0e8-258aa1e1a4a5-hosts-file\") pod \"node-resolver-7jq9m\" (UID: \"2d258432-7e24-4b20-a0e8-258aa1e1a4a5\") " pod="openshift-dns/node-resolver-7jq9m" Oct 30 18:33:19.835542 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.835455 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-etc-kubernetes\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.835542 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.835473 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-node-log\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.835542 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.835508 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02de5f04-abcb-440e-8e9e-55d41cd543e2-host\") pod \"node-ca-7c9gt\" (UID: \"02de5f04-abcb-440e-8e9e-55d41cd543e2\") " pod="openshift-image-registry/node-ca-7c9gt" Oct 30 18:33:19.835542 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.835526 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-sysconfig\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.835745 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.835582 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-sysctl-d\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.835745 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.835619 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-var-lib-kubelet\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.835745 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.835647 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-host\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.835745 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.835677 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r59q\" (UniqueName: \"kubernetes.io/projected/c1ea54d3-410e-48cf-80f6-46be7218f433-kube-api-access-8r59q\") pod \"iptables-alerter-qnqxv\" (UID: \"c1ea54d3-410e-48cf-80f6-46be7218f433\") " pod="openshift-network-operator/iptables-alerter-qnqxv" Oct 30 18:33:19.835745 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.835702 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/429722d3-91b1-4c92-b78e-19a12bc5bac9-dbus\") pod \"global-pull-secret-syncer-frkss\" (UID: \"429722d3-91b1-4c92-b78e-19a12bc5bac9\") " pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:19.835745 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.835728 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-kubelet\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.835996 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.835754 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e9a334f2-12b2-4188-8bfa-7bc69559f024-env-overrides\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.835996 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.835781 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpw2g\" (UniqueName: \"kubernetes.io/projected/3b5808a2-cbcf-4f2d-a401-191f429cc6be-kube-api-access-zpw2g\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.835996 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.835822 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1ea54d3-410e-48cf-80f6-46be7218f433-iptables-alerter-script\") pod \"iptables-alerter-qnqxv\" (UID: \"c1ea54d3-410e-48cf-80f6-46be7218f433\") " pod="openshift-network-operator/iptables-alerter-qnqxv" Oct 30 18:33:19.835996 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.835868 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/002053ca-649b-4a56-9ffd-25406d82fe99-agent-certs\") pod \"konnectivity-agent-sd88x\" (UID: \"002053ca-649b-4a56-9ffd-25406d82fe99\") " pod="kube-system/konnectivity-agent-sd88x" Oct 30 18:33:19.835996 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.835889 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-cnibin\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:33:19.835996 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.835904 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-systemd\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.835996 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.835920 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-sys\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.835996 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.835958 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-multus-cni-dir\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.836342 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836000 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-run-netns\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.836342 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836038 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-socket-dir\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:33:19.836342 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836066 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84bbebc-7b6d-4e22-aacd-c98586717109-metrics-certs\") pod \"network-metrics-daemon-flgwv\" (UID: \"c84bbebc-7b6d-4e22-aacd-c98586717109\") " pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:19.836342 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836100 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zdng\" (UniqueName: \"kubernetes.io/projected/2d258432-7e24-4b20-a0e8-258aa1e1a4a5-kube-api-access-2zdng\") pod \"node-resolver-7jq9m\" (UID: \"2d258432-7e24-4b20-a0e8-258aa1e1a4a5\") " pod="openshift-dns/node-resolver-7jq9m" Oct 30 18:33:19.836342 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836135 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/429722d3-91b1-4c92-b78e-19a12bc5bac9-kubelet-config\") pod \"global-pull-secret-syncer-frkss\" (UID: \"429722d3-91b1-4c92-b78e-19a12bc5bac9\") " pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:19.836342 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836160 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-run\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.836342 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836185 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-hostroot\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.836342 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836217 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-device-dir\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:33:19.836342 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836260 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e9a334f2-12b2-4188-8bfa-7bc69559f024-ovnkube-config\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.836342 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836281 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmg24\" (UniqueName: \"kubernetes.io/projected/02de5f04-abcb-440e-8e9e-55d41cd543e2-kube-api-access-nmg24\") pod \"node-ca-7c9gt\" (UID: \"02de5f04-abcb-440e-8e9e-55d41cd543e2\") " pod="openshift-image-registry/node-ca-7c9gt" Oct 30 18:33:19.836342 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836299 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-modprobe-d\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.836342 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836314 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d676d57c-8007-406d-8b45-e3a6c33ac3c9-cni-binary-copy\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.836342 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836328 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-var-lib-kubelet\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.836342 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836342 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d676d57c-8007-406d-8b45-e3a6c33ac3c9-multus-daemon-config\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.836948 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836366 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2wzg\" (UniqueName: \"kubernetes.io/projected/c84bbebc-7b6d-4e22-aacd-c98586717109-kube-api-access-w2wzg\") pod \"network-metrics-daemon-flgwv\" (UID: \"c84bbebc-7b6d-4e22-aacd-c98586717109\") " pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:19.836948 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836401 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-cni-bin\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.836948 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836423 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-system-cni-dir\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.836948 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836439 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-slash\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.836948 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836453 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:33:19.836948 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836510 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:33:19.836948 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836543 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-lib-modules\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.836948 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836567 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfbp8\" (UniqueName: \"kubernetes.io/projected/d676d57c-8007-406d-8b45-e3a6c33ac3c9-kube-api-access-cfbp8\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.836948 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836592 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-run-openvswitch\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.836948 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836617 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/02de5f04-abcb-440e-8e9e-55d41cd543e2-serviceca\") pod \"node-ca-7c9gt\" (UID: \"02de5f04-abcb-440e-8e9e-55d41cd543e2\") " pod="openshift-image-registry/node-ca-7c9gt" Oct 30 18:33:19.836948 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836660 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrj48\" (UniqueName: \"kubernetes.io/projected/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-kube-api-access-vrj48\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:33:19.836948 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836677 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-kubernetes\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.836948 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836691 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-tuned\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.836948 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836705 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-multus-conf-dir\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.836948 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836719 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-registration-dir\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:33:19.836948 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836733 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-run-ovn-kubernetes\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.837592 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836763 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-cni-netd\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.837592 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836842 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e9a334f2-12b2-4188-8bfa-7bc69559f024-ovn-node-metrics-cert\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.837592 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836865 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:33:19.837592 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836886 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-var-lib-cni-multus\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.837592 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836903 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-run-multus-certs\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.837592 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836923 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxxl5\" (UniqueName: \"kubernetes.io/projected/17dae92a-ae2b-4c09-8001-48a81a8cf809-kube-api-access-mxxl5\") pod \"network-check-target-hh4r5\" (UID: \"17dae92a-ae2b-4c09-8001-48a81a8cf809\") " pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:19.837592 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836942 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1ea54d3-410e-48cf-80f6-46be7218f433-host-slash\") pod \"iptables-alerter-qnqxv\" (UID: \"c1ea54d3-410e-48cf-80f6-46be7218f433\") " pod="openshift-network-operator/iptables-alerter-qnqxv" Oct 30 18:33:19.837592 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836959 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-etc-selinux\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:33:19.837592 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.836974 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrf7w\" (UniqueName: \"kubernetes.io/projected/e9a334f2-12b2-4188-8bfa-7bc69559f024-kube-api-access-mrf7w\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.837592 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.837011 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qxnh\" (UniqueName: \"kubernetes.io/projected/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-kube-api-access-4qxnh\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:33:19.837592 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.837048 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-run-k8s-cni-cncf-io\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.837592 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.837074 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:33:19.837592 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.837093 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-run-systemd\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.837592 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.837110 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.837592 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.837126 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e9a334f2-12b2-4188-8bfa-7bc69559f024-ovnkube-script-lib\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.837592 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.837141 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b5808a2-cbcf-4f2d-a401-191f429cc6be-tmp\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.838026 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.837159 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-var-lib-cni-bin\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.838026 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.837175 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-sys-fs\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:33:19.838026 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.837189 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-etc-openvswitch\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.838026 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.837202 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-run-ovn\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.838026 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.837215 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-log-socket\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.838026 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.837229 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-cni-binary-copy\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:33:19.838026 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.837244 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d258432-7e24-4b20-a0e8-258aa1e1a4a5-tmp-dir\") pod \"node-resolver-7jq9m\" (UID: \"2d258432-7e24-4b20-a0e8-258aa1e1a4a5\") " pod="openshift-dns/node-resolver-7jq9m" Oct 30 18:33:19.838026 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.837287 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-system-cni-dir\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:33:19.838026 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.837318 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-var-lib-openvswitch\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.838026 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.837343 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-sysctl-conf\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.838026 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.837365 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-cnibin\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.838026 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.837391 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-multus-socket-dir-parent\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.838026 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.837413 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/429722d3-91b1-4c92-b78e-19a12bc5bac9-original-pull-secret\") pod \"global-pull-secret-syncer-frkss\" (UID: \"429722d3-91b1-4c92-b78e-19a12bc5bac9\") " pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:19.838026 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.837428 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-systemd-units\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.838026 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.837469 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-run-netns\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.838026 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.837522 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/002053ca-649b-4a56-9ffd-25406d82fe99-konnectivity-ca\") pod \"konnectivity-agent-sd88x\" (UID: \"002053ca-649b-4a56-9ffd-25406d82fe99\") " pod="kube-system/konnectivity-agent-sd88x" Oct 30 18:33:19.875205 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.875169 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nh9hp\"" Oct 30 18:33:19.875205 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.875202 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:33:19.879262 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.879231 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cr7s6\"" Oct 30 18:33:19.879422 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.879321 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-g6mvb\"" Oct 30 18:33:19.879422 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.879318 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4bpx7\"" Oct 30 18:33:19.880151 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.880132 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 18:33:19.880151 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.880152 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 18:33:19.880642 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.880629 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 18:33:19.881206 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.881193 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cgz7p\"" Oct 30 18:33:19.881409 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.881395 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 18:33:19.881485 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.881414 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 18:33:19.881485 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.881439 2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 18:33:19.881485 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.881447 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 18:33:19.881485 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.881469 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 18:33:19.881666 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.881559 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 18:33:19.881666 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.881560 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rkt2d" Oct 30 18:33:19.881666 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.881572 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 18:33:19.881666 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.881618 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 18:33:19.938288 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938250 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e9a334f2-12b2-4188-8bfa-7bc69559f024-env-overrides\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.938288 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938291 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpw2g\" (UniqueName: \"kubernetes.io/projected/3b5808a2-cbcf-4f2d-a401-191f429cc6be-kube-api-access-zpw2g\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.938457 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938309 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1ea54d3-410e-48cf-80f6-46be7218f433-iptables-alerter-script\") pod \"iptables-alerter-qnqxv\" (UID: \"c1ea54d3-410e-48cf-80f6-46be7218f433\") " pod="openshift-network-operator/iptables-alerter-qnqxv" Oct 30 18:33:19.938457 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938332 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/002053ca-649b-4a56-9ffd-25406d82fe99-agent-certs\") pod \"konnectivity-agent-sd88x\" (UID: \"002053ca-649b-4a56-9ffd-25406d82fe99\") " pod="kube-system/konnectivity-agent-sd88x" Oct 30 18:33:19.938560 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938473 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-cnibin\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:33:19.938560 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938534 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-systemd\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.938651 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938558 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-sys\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.938651 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938581 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-multus-cni-dir\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.938651 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938608 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-run-netns\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.938651 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938631 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-socket-dir\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:33:19.938839 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938656 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84bbebc-7b6d-4e22-aacd-c98586717109-metrics-certs\") pod \"network-metrics-daemon-flgwv\" (UID: \"c84bbebc-7b6d-4e22-aacd-c98586717109\") " pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:19.938839 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938680 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zdng\" (UniqueName: \"kubernetes.io/projected/2d258432-7e24-4b20-a0e8-258aa1e1a4a5-kube-api-access-2zdng\") pod \"node-resolver-7jq9m\" (UID: \"2d258432-7e24-4b20-a0e8-258aa1e1a4a5\") " pod="openshift-dns/node-resolver-7jq9m" Oct 30 18:33:19.938839 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938707 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/429722d3-91b1-4c92-b78e-19a12bc5bac9-kubelet-config\") pod \"global-pull-secret-syncer-frkss\" (UID: \"429722d3-91b1-4c92-b78e-19a12bc5bac9\") " pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:19.938839 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938708 2571 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 18:33:19.938839 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938723 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-run-netns\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.938839 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938729 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-run\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.938839 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938759 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-sys\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.938839 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938780 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-hostroot\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.938839 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938798 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e9a334f2-12b2-4188-8bfa-7bc69559f024-env-overrides\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.938839 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938810 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-device-dir\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:33:19.938839 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938677 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-systemd\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.939348 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938838 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e9a334f2-12b2-4188-8bfa-7bc69559f024-ovnkube-config\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.939348 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938854 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-run\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.939348 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938877 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-multus-cni-dir\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.939348 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938878 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmg24\" (UniqueName: \"kubernetes.io/projected/02de5f04-abcb-440e-8e9e-55d41cd543e2-kube-api-access-nmg24\") pod \"node-ca-7c9gt\" (UID: \"02de5f04-abcb-440e-8e9e-55d41cd543e2\") " pod="openshift-image-registry/node-ca-7c9gt" Oct 30 18:33:19.939348 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938914 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/429722d3-91b1-4c92-b78e-19a12bc5bac9-kubelet-config\") pod \"global-pull-secret-syncer-frkss\" (UID: \"429722d3-91b1-4c92-b78e-19a12bc5bac9\") " pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:19.939348 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938920 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-modprobe-d\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.939348 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938934 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1ea54d3-410e-48cf-80f6-46be7218f433-iptables-alerter-script\") pod \"iptables-alerter-qnqxv\" (UID: \"c1ea54d3-410e-48cf-80f6-46be7218f433\") " pod="openshift-network-operator/iptables-alerter-qnqxv" Oct 30 18:33:19.939348 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938950 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d676d57c-8007-406d-8b45-e3a6c33ac3c9-cni-binary-copy\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.939348 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938975 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-var-lib-kubelet\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.939348 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939017 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-socket-dir\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:33:19.939348 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939032 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d676d57c-8007-406d-8b45-e3a6c33ac3c9-multus-daemon-config\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.939348 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939064 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-var-lib-kubelet\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.939348 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939070 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2wzg\" (UniqueName: \"kubernetes.io/projected/c84bbebc-7b6d-4e22-aacd-c98586717109-kube-api-access-w2wzg\") pod \"network-metrics-daemon-flgwv\" (UID: \"c84bbebc-7b6d-4e22-aacd-c98586717109\") " pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:19.939348 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939104 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-cni-bin\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.939348 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939114 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-device-dir\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:33:19.939348 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939128 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-system-cni-dir\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.939348 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939139 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-modprobe-d\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.939348 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:19.939039 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:33:19.940175 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939191 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-slash\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.940175 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939196 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-cni-bin\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.940175 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939153 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-slash\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.940175 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939191 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-system-cni-dir\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.940175 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939071 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-hostroot\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.940175 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:19.939261 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c84bbebc-7b6d-4e22-aacd-c98586717109-metrics-certs podName:c84bbebc-7b6d-4e22-aacd-c98586717109 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:20.439217874 +0000 UTC m=+2.291563706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c84bbebc-7b6d-4e22-aacd-c98586717109-metrics-certs") pod "network-metrics-daemon-flgwv" (UID: "c84bbebc-7b6d-4e22-aacd-c98586717109") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:33:19.940175 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939279 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:33:19.940175 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939308 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:33:19.940175 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939331 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-lib-modules\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.940175 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939341 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e9a334f2-12b2-4188-8bfa-7bc69559f024-ovnkube-config\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.940175 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939355 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfbp8\" (UniqueName: \"kubernetes.io/projected/d676d57c-8007-406d-8b45-e3a6c33ac3c9-kube-api-access-cfbp8\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.940175 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939392 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-run-openvswitch\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.940175 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939418 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/02de5f04-abcb-440e-8e9e-55d41cd543e2-serviceca\") pod \"node-ca-7c9gt\" (UID: \"02de5f04-abcb-440e-8e9e-55d41cd543e2\") " pod="openshift-image-registry/node-ca-7c9gt" Oct 30 18:33:19.940175 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939446 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrj48\" (UniqueName: \"kubernetes.io/projected/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-kube-api-access-vrj48\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:33:19.940175 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939471 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-kubernetes\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.940175 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939510 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d676d57c-8007-406d-8b45-e3a6c33ac3c9-cni-binary-copy\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.940175 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939521 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-tuned\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.940999 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939553 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-multus-conf-dir\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.940999 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939577 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-registration-dir\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:33:19.940999 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939599 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-run-ovn-kubernetes\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.940999 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939620 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-cni-netd\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.940999 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939639 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e9a334f2-12b2-4188-8bfa-7bc69559f024-ovn-node-metrics-cert\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.940999 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939647 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-lib-modules\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.940999 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939660 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:33:19.940999 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939690 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-var-lib-cni-multus\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.940999 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939718 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-run-multus-certs\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.940999 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939742 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxxl5\" (UniqueName: \"kubernetes.io/projected/17dae92a-ae2b-4c09-8001-48a81a8cf809-kube-api-access-mxxl5\") pod \"network-check-target-hh4r5\" (UID: \"17dae92a-ae2b-4c09-8001-48a81a8cf809\") " pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:19.940999 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939779 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-var-lib-cni-multus\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.940999 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939807 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1ea54d3-410e-48cf-80f6-46be7218f433-host-slash\") pod \"iptables-alerter-qnqxv\" (UID: \"c1ea54d3-410e-48cf-80f6-46be7218f433\") " pod="openshift-network-operator/iptables-alerter-qnqxv" Oct 30 18:33:19.940999 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939826 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-etc-selinux\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:33:19.940999 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939841 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrf7w\" (UniqueName: \"kubernetes.io/projected/e9a334f2-12b2-4188-8bfa-7bc69559f024-kube-api-access-mrf7w\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.940999 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.938710 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-cnibin\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:33:19.940999 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939865 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qxnh\" (UniqueName: \"kubernetes.io/projected/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-kube-api-access-4qxnh\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:33:19.940999 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939898 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-run-k8s-cni-cncf-io\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.942087 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939926 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-run-openvswitch\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.942087 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.939971 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-kubernetes\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.942087 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940021 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-run-ovn-kubernetes\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.942087 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940031 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-multus-conf-dir\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.942087 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940066 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:33:19.942087 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940068 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-registration-dir\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:33:19.942087 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940089 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-run-systemd\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.942087 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940114 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-run-k8s-cni-cncf-io\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.942087 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940138 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-cni-netd\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.942087 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940113 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.942087 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940200 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:33:19.942087 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940210 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e9a334f2-12b2-4188-8bfa-7bc69559f024-ovnkube-script-lib\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.942087 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940243 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b5808a2-cbcf-4f2d-a401-191f429cc6be-tmp\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.942087 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940271 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-var-lib-cni-bin\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.942087 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940301 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-sys-fs\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:33:19.942087 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940329 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-etc-openvswitch\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.942087 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940356 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-run-ovn\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.942993 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940358 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d676d57c-8007-406d-8b45-e3a6c33ac3c9-multus-daemon-config\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.942993 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940377 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-run-systemd\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.942993 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940382 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-log-socket\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.942993 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940424 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-var-lib-cni-bin\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.942993 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940454 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-cni-binary-copy\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:33:19.942993 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940458 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-sys-fs\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:33:19.942993 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940472 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-log-socket\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.942993 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940514 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-etc-openvswitch\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.942993 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940547 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d258432-7e24-4b20-a0e8-258aa1e1a4a5-tmp-dir\") pod \"node-resolver-7jq9m\" (UID: \"2d258432-7e24-4b20-a0e8-258aa1e1a4a5\") " pod="openshift-dns/node-resolver-7jq9m" Oct 30 18:33:19.942993 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940573 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-etc-selinux\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:33:19.942993 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940576 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-system-cni-dir\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:33:19.942993 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940611 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-system-cni-dir\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:33:19.942993 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940628 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-var-lib-openvswitch\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.942993 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940655 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-run-ovn\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.942993 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940657 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-sysctl-conf\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.942993 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940658 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-run-multus-certs\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.942993 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940700 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-cnibin\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.942993 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940723 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-var-lib-openvswitch\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.943763 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940724 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-multus-socket-dir-parent\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.943763 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940760 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/429722d3-91b1-4c92-b78e-19a12bc5bac9-original-pull-secret\") pod \"global-pull-secret-syncer-frkss\" (UID: \"429722d3-91b1-4c92-b78e-19a12bc5bac9\") " pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:19.943763 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940784 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e9a334f2-12b2-4188-8bfa-7bc69559f024-ovnkube-script-lib\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.943763 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940799 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-systemd-units\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.943763 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940765 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-multus-socket-dir-parent\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.943763 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940827 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-run-netns\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.943763 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940878 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.943763 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940924 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-cnibin\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.943763 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940954 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/002053ca-649b-4a56-9ffd-25406d82fe99-konnectivity-ca\") pod \"konnectivity-agent-sd88x\" (UID: \"002053ca-649b-4a56-9ffd-25406d82fe99\") " pod="kube-system/konnectivity-agent-sd88x" Oct 30 18:33:19.943763 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940964 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d258432-7e24-4b20-a0e8-258aa1e1a4a5-tmp-dir\") pod \"node-resolver-7jq9m\" (UID: \"2d258432-7e24-4b20-a0e8-258aa1e1a4a5\") " pod="openshift-dns/node-resolver-7jq9m" Oct 30 18:33:19.943763 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940982 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d258432-7e24-4b20-a0e8-258aa1e1a4a5-hosts-file\") pod \"node-resolver-7jq9m\" (UID: \"2d258432-7e24-4b20-a0e8-258aa1e1a4a5\") " pod="openshift-dns/node-resolver-7jq9m" Oct 30 18:33:19.943763 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.940950 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-sysctl-conf\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.943763 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.941043 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-etc-kubernetes\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.943763 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.941051 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-systemd-units\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.943763 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.941007 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-etc-kubernetes\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:19.943763 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.941098 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d258432-7e24-4b20-a0e8-258aa1e1a4a5-hosts-file\") pod \"node-resolver-7jq9m\" (UID: \"2d258432-7e24-4b20-a0e8-258aa1e1a4a5\") " pod="openshift-dns/node-resolver-7jq9m" Oct 30 18:33:19.943763 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.941109 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-node-log\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.943763 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:19.941121 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:33:19.944229 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.941131 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-run-netns\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.944229 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.941171 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02de5f04-abcb-440e-8e9e-55d41cd543e2-host\") pod \"node-ca-7c9gt\" (UID: \"02de5f04-abcb-440e-8e9e-55d41cd543e2\") " pod="openshift-image-registry/node-ca-7c9gt" Oct 30 18:33:19.944229 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.941200 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:33:19.944229 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.941224 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-sysconfig\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.944229 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.941236 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02de5f04-abcb-440e-8e9e-55d41cd543e2-host\") pod \"node-ca-7c9gt\" (UID: \"02de5f04-abcb-440e-8e9e-55d41cd543e2\") " pod="openshift-image-registry/node-ca-7c9gt" Oct 30 18:33:19.944229 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:19.941287 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429722d3-91b1-4c92-b78e-19a12bc5bac9-original-pull-secret podName:429722d3-91b1-4c92-b78e-19a12bc5bac9 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:20.441269448 +0000 UTC m=+2.293615260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/429722d3-91b1-4c92-b78e-19a12bc5bac9-original-pull-secret") pod "global-pull-secret-syncer-frkss" (UID: "429722d3-91b1-4c92-b78e-19a12bc5bac9") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:33:19.944229 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.941293 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:33:19.944229 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.941313 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-cni-binary-copy\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:33:19.944229 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.941337 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-sysctl-d\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.944229 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.941355 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-node-log\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.944229 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.941364 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-var-lib-kubelet\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.944229 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.941387 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-sysconfig\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.944229 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.941420 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-var-lib-kubelet\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.944229 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.941423 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-host\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.944229 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.941440 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-sysctl-d\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.944229 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.941469 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r59q\" (UniqueName: \"kubernetes.io/projected/c1ea54d3-410e-48cf-80f6-46be7218f433-kube-api-access-8r59q\") pod \"iptables-alerter-qnqxv\" (UID: \"c1ea54d3-410e-48cf-80f6-46be7218f433\") " pod="openshift-network-operator/iptables-alerter-qnqxv" Oct 30 18:33:19.944229 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.941471 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-host\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.944702 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.941482 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1ea54d3-410e-48cf-80f6-46be7218f433-host-slash\") pod \"iptables-alerter-qnqxv\" (UID: \"c1ea54d3-410e-48cf-80f6-46be7218f433\") " pod="openshift-network-operator/iptables-alerter-qnqxv" Oct 30 18:33:19.944702 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.941554 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/429722d3-91b1-4c92-b78e-19a12bc5bac9-dbus\") pod \"global-pull-secret-syncer-frkss\" (UID: \"429722d3-91b1-4c92-b78e-19a12bc5bac9\") " pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:19.944702 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.941622 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-kubelet\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.944702 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.941735 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-kubelet\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.944702 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.941747 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/429722d3-91b1-4c92-b78e-19a12bc5bac9-dbus\") pod \"global-pull-secret-syncer-frkss\" (UID: \"429722d3-91b1-4c92-b78e-19a12bc5bac9\") " pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:19.944702 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.942815 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:33:19.944702 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.942848 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/02de5f04-abcb-440e-8e9e-55d41cd543e2-serviceca\") pod \"node-ca-7c9gt\" (UID: \"02de5f04-abcb-440e-8e9e-55d41cd543e2\") " pod="openshift-image-registry/node-ca-7c9gt" Oct 30 18:33:19.944702 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.942995 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/002053ca-649b-4a56-9ffd-25406d82fe99-konnectivity-ca\") pod \"konnectivity-agent-sd88x\" (UID: \"002053ca-649b-4a56-9ffd-25406d82fe99\") " pod="kube-system/konnectivity-agent-sd88x" Oct 30 18:33:19.944702 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.943112 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-tuned\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:19.944702 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.943211 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e9a334f2-12b2-4188-8bfa-7bc69559f024-ovn-node-metrics-cert\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:19.944702 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.943288 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/002053ca-649b-4a56-9ffd-25406d82fe99-agent-certs\") pod \"konnectivity-agent-sd88x\" (UID: \"002053ca-649b-4a56-9ffd-25406d82fe99\") " pod="kube-system/konnectivity-agent-sd88x" Oct 30 18:33:19.944702 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:19.943685 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b5808a2-cbcf-4f2d-a401-191f429cc6be-tmp\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:21.406168 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:21.406116 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpw2g\" (UniqueName: \"kubernetes.io/projected/3b5808a2-cbcf-4f2d-a401-191f429cc6be-kube-api-access-zpw2g\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:21.406757 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:21.406679 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:33:21.406757 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:21.406710 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:33:21.406757 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:21.406725 2571 projected.go:196] Error preparing data for projected volume kube-api-access-mxxl5 for pod openshift-network-diagnostics/network-check-target-hh4r5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:33:21.406986 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:21.406840 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17dae92a-ae2b-4c09-8001-48a81a8cf809-kube-api-access-mxxl5 podName:17dae92a-ae2b-4c09-8001-48a81a8cf809 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:21.906819099 +0000 UTC m=+2.354527782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mxxl5" (UniqueName: "kubernetes.io/projected/17dae92a-ae2b-4c09-8001-48a81a8cf809-kube-api-access-mxxl5") pod "network-check-target-hh4r5" (UID: "17dae92a-ae2b-4c09-8001-48a81a8cf809") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:33:21.411973 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:21.411942 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfbp8\" (UniqueName: \"kubernetes.io/projected/d676d57c-8007-406d-8b45-e3a6c33ac3c9-kube-api-access-cfbp8\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:33:21.411973 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:21.411966 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zdng\" (UniqueName: \"kubernetes.io/projected/2d258432-7e24-4b20-a0e8-258aa1e1a4a5-kube-api-access-2zdng\") pod \"node-resolver-7jq9m\" (UID: \"2d258432-7e24-4b20-a0e8-258aa1e1a4a5\") " pod="openshift-dns/node-resolver-7jq9m" Oct 30 18:33:21.412200 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:21.412016 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmg24\" (UniqueName: \"kubernetes.io/projected/02de5f04-abcb-440e-8e9e-55d41cd543e2-kube-api-access-nmg24\") pod \"node-ca-7c9gt\" (UID: \"02de5f04-abcb-440e-8e9e-55d41cd543e2\") " pod="openshift-image-registry/node-ca-7c9gt" Oct 30 18:33:21.412200 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:21.412038 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2wzg\" (UniqueName: \"kubernetes.io/projected/c84bbebc-7b6d-4e22-aacd-c98586717109-kube-api-access-w2wzg\") pod \"network-metrics-daemon-flgwv\" (UID: \"c84bbebc-7b6d-4e22-aacd-c98586717109\") " pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:21.414761 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:21.414725 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r59q\" (UniqueName: \"kubernetes.io/projected/c1ea54d3-410e-48cf-80f6-46be7218f433-kube-api-access-8r59q\") pod \"iptables-alerter-qnqxv\" (UID: \"c1ea54d3-410e-48cf-80f6-46be7218f433\") " pod="openshift-network-operator/iptables-alerter-qnqxv" Oct 30 18:33:21.414924 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:21.414766 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrf7w\" (UniqueName: \"kubernetes.io/projected/e9a334f2-12b2-4188-8bfa-7bc69559f024-kube-api-access-mrf7w\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:21.414995 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:21.414950 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrj48\" (UniqueName: \"kubernetes.io/projected/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-kube-api-access-vrj48\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:33:21.414995 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:21.414976 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qxnh\" (UniqueName: \"kubernetes.io/projected/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-kube-api-access-4qxnh\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:33:21.508392 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:21.508351 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:21.520882 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:21.520837 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:33:21.535280 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:21.535246 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7c9gt" Oct 30 18:33:21.541454 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:21.541418 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f2745" Oct 30 18:33:21.566244 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:21.566208 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qnqxv" Oct 30 18:33:21.585620 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:21.585581 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:33:21.599586 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:21.599554 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sd88x" Oct 30 18:33:21.608332 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:21.608282 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea9d11f9_e0cb_40f4_ad8e_7925bdcdd096.slice/crio-f09c09059ed4098f4965821046cc526993b532bc4c4ada49da6a8f4c06ef1984 WatchSource:0}: Error finding container f09c09059ed4098f4965821046cc526993b532bc4c4ada49da6a8f4c06ef1984: Status 404 returned error can't find the container with id f09c09059ed4098f4965821046cc526993b532bc4c4ada49da6a8f4c06ef1984 Oct 30 18:33:21.608941 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:21.608913 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd156aeb8e5a3750f522b65c4725c3f82.slice/crio-52378ab0946742667c5f06bbacbc5584f1402600bd3ed27f3d6cf257ec8643a1 WatchSource:0}: Error finding container 52378ab0946742667c5f06bbacbc5584f1402600bd3ed27f3d6cf257ec8643a1: Status 404 returned error can't find the container with id 52378ab0946742667c5f06bbacbc5584f1402600bd3ed27f3d6cf257ec8643a1 Oct 30 18:33:21.611313 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:21.611274 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod002053ca_649b_4a56_9ffd_25406d82fe99.slice/crio-d3c2f17fd9744b24d93ee97a8a4df78541239379fbe521b0c90f98f33fc84c46 WatchSource:0}: Error finding container d3c2f17fd9744b24d93ee97a8a4df78541239379fbe521b0c90f98f33fc84c46: Status 404 returned error can't find the container with id d3c2f17fd9744b24d93ee97a8a4df78541239379fbe521b0c90f98f33fc84c46 Oct 30 18:33:21.611765 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:21.611747 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7jq9m" Oct 30 18:33:21.611972 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:21.611954 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ae57dd60561f1d7425076e8b67e7e3.slice/crio-b758f360492f15fcb6ca783e0055106b3c2b56d740b656e4fe8f3c47eafafae8 WatchSource:0}: Error finding container b758f360492f15fcb6ca783e0055106b3c2b56d740b656e4fe8f3c47eafafae8: Status 404 returned error can't find the container with id b758f360492f15fcb6ca783e0055106b3c2b56d740b656e4fe8f3c47eafafae8 Oct 30 18:33:21.614887 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:21.614875 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:33:21.620420 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:21.620400 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d258432_7e24_4b20_a0e8_258aa1e1a4a5.slice/crio-6bd1ea4b68f21ee96169e822f835d9eda8fa7cd8d83e3a2e701ea5778d166ee6 WatchSource:0}: Error finding container 6bd1ea4b68f21ee96169e822f835d9eda8fa7cd8d83e3a2e701ea5778d166ee6: Status 404 returned error can't find the container with id 6bd1ea4b68f21ee96169e822f835d9eda8fa7cd8d83e3a2e701ea5778d166ee6 Oct 30 18:33:21.623798 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:21.623778 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:33:21.848817 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:21.848780 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/429722d3-91b1-4c92-b78e-19a12bc5bac9-original-pull-secret\") pod \"global-pull-secret-syncer-frkss\" (UID: \"429722d3-91b1-4c92-b78e-19a12bc5bac9\") " pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:21.848817 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:21.848824 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84bbebc-7b6d-4e22-aacd-c98586717109-metrics-certs\") pod \"network-metrics-daemon-flgwv\" (UID: \"c84bbebc-7b6d-4e22-aacd-c98586717109\") " pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:21.849019 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:21.848936 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:33:21.849019 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:21.848937 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:33:21.849019 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:21.848987 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c84bbebc-7b6d-4e22-aacd-c98586717109-metrics-certs podName:c84bbebc-7b6d-4e22-aacd-c98586717109 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:22.848974988 +0000 UTC m=+3.296683651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c84bbebc-7b6d-4e22-aacd-c98586717109-metrics-certs") pod "network-metrics-daemon-flgwv" (UID: "c84bbebc-7b6d-4e22-aacd-c98586717109") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:33:21.849019 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:21.849001 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429722d3-91b1-4c92-b78e-19a12bc5bac9-original-pull-secret podName:429722d3-91b1-4c92-b78e-19a12bc5bac9 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:22.848993738 +0000 UTC m=+3.296702402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/429722d3-91b1-4c92-b78e-19a12bc5bac9-original-pull-secret") pod "global-pull-secret-syncer-frkss" (UID: "429722d3-91b1-4c92-b78e-19a12bc5bac9") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:33:21.949891 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:21.949853 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxxl5\" (UniqueName: \"kubernetes.io/projected/17dae92a-ae2b-4c09-8001-48a81a8cf809-kube-api-access-mxxl5\") pod \"network-check-target-hh4r5\" (UID: \"17dae92a-ae2b-4c09-8001-48a81a8cf809\") " pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:21.950511 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:21.950025 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:33:21.950511 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:21.950046 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:33:21.950511 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:21.950057 2571 projected.go:196] Error preparing data for projected volume kube-api-access-mxxl5 for pod openshift-network-diagnostics/network-check-target-hh4r5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:33:21.950511 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:21.950109 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17dae92a-ae2b-4c09-8001-48a81a8cf809-kube-api-access-mxxl5 podName:17dae92a-ae2b-4c09-8001-48a81a8cf809 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:22.950095718 +0000 UTC m=+3.397804381 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mxxl5" (UniqueName: "kubernetes.io/projected/17dae92a-ae2b-4c09-8001-48a81a8cf809-kube-api-access-mxxl5") pod "network-check-target-hh4r5" (UID: "17dae92a-ae2b-4c09-8001-48a81a8cf809") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:33:21.970845 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:21.970809 2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:33:22.074282 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:22.074220 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxpzg" event={"ID":"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89","Type":"ContainerStarted","Data":"5e5badf7e5aa75ed05a8c0fb739b661376477138c4bc5b96ac1d65f827d17426"} Oct 30 18:33:22.075219 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:22.075192 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7jq9m" event={"ID":"2d258432-7e24-4b20-a0e8-258aa1e1a4a5","Type":"ContainerStarted","Data":"6bd1ea4b68f21ee96169e822f835d9eda8fa7cd8d83e3a2e701ea5778d166ee6"} Oct 30 18:33:22.076247 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:22.076222 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal" event={"ID":"76ae57dd60561f1d7425076e8b67e7e3","Type":"ContainerStarted","Data":"b758f360492f15fcb6ca783e0055106b3c2b56d740b656e4fe8f3c47eafafae8"} Oct 30 18:33:22.077351 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:22.077329 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sd88x" event={"ID":"002053ca-649b-4a56-9ffd-25406d82fe99","Type":"ContainerStarted","Data":"d3c2f17fd9744b24d93ee97a8a4df78541239379fbe521b0c90f98f33fc84c46"} Oct 30 18:33:22.078472 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:22.078442 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" event={"ID":"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096","Type":"ContainerStarted","Data":"f09c09059ed4098f4965821046cc526993b532bc4c4ada49da6a8f4c06ef1984"} Oct 30 18:33:22.079453 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:22.079433 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal" event={"ID":"d156aeb8e5a3750f522b65c4725c3f82","Type":"ContainerStarted","Data":"52378ab0946742667c5f06bbacbc5584f1402600bd3ed27f3d6cf257ec8643a1"} Oct 30 18:33:22.288346 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:22.288291 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:28:21 +0000 UTC" deadline="2028-06-21 01:26:10.871839952 +0000 UTC" Oct 30 18:33:22.288346 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:22.288337 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14382h52m48.583505645s" Oct 30 18:33:22.326600 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:22.326569 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a334f2_12b2_4188_8bfa_7bc69559f024.slice/crio-df73aee8aeec5eb2ccaf7e6e156a0cd993c9e672c36dd57025e1966f698b1637 WatchSource:0}: Error finding container df73aee8aeec5eb2ccaf7e6e156a0cd993c9e672c36dd57025e1966f698b1637: Status 404 returned error can't find the container with id df73aee8aeec5eb2ccaf7e6e156a0cd993c9e672c36dd57025e1966f698b1637 Oct 30 18:33:22.332452 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:22.332425 2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:33:22.336436 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:22.336401 2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:33:22.370815 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:22.370786 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5808a2_cbcf_4f2d_a401_191f429cc6be.slice/crio-3f796e66db196ea76ef51896a297e905fe8e8216b18fd5a23362c08c5f2df32f WatchSource:0}: Error finding container 3f796e66db196ea76ef51896a297e905fe8e8216b18fd5a23362c08c5f2df32f: Status 404 returned error can't find the container with id 3f796e66db196ea76ef51896a297e905fe8e8216b18fd5a23362c08c5f2df32f Oct 30 18:33:22.403498 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:22.403457 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ea54d3_410e_48cf_80f6_46be7218f433.slice/crio-825ca0cf96a8bbb51b048ef97bf527945b170c242b223a8d2f73c006a5fb0c9b WatchSource:0}: Error finding container 825ca0cf96a8bbb51b048ef97bf527945b170c242b223a8d2f73c006a5fb0c9b: Status 404 returned error can't find the container with id 825ca0cf96a8bbb51b048ef97bf527945b170c242b223a8d2f73c006a5fb0c9b Oct 30 18:33:22.427274 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:22.427243 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd676d57c_8007_406d_8b45_e3a6c33ac3c9.slice/crio-3df699b0fae271fb197e9b9b3b127585731af8cb1f09d00cb1a0b4368a49a505 WatchSource:0}: Error finding container 3df699b0fae271fb197e9b9b3b127585731af8cb1f09d00cb1a0b4368a49a505: Status 404 returned error can't find the container with id 3df699b0fae271fb197e9b9b3b127585731af8cb1f09d00cb1a0b4368a49a505 Oct 30 18:33:22.427773 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:22.427755 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02de5f04_abcb_440e_8e9e_55d41cd543e2.slice/crio-879e92e7e28f8539194240f70c7e058330b43243540df67b153118eeab2bda0f WatchSource:0}: Error finding container 879e92e7e28f8539194240f70c7e058330b43243540df67b153118eeab2bda0f: Status 404 returned error can't find the container with id 879e92e7e28f8539194240f70c7e058330b43243540df67b153118eeab2bda0f Oct 30 18:33:22.856591 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:22.856186 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/429722d3-91b1-4c92-b78e-19a12bc5bac9-original-pull-secret\") pod \"global-pull-secret-syncer-frkss\" (UID: \"429722d3-91b1-4c92-b78e-19a12bc5bac9\") " pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:22.856591 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:22.856247 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84bbebc-7b6d-4e22-aacd-c98586717109-metrics-certs\") pod \"network-metrics-daemon-flgwv\" (UID: \"c84bbebc-7b6d-4e22-aacd-c98586717109\") " pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:22.856591 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:22.856377 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:33:22.856591 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:22.856441 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c84bbebc-7b6d-4e22-aacd-c98586717109-metrics-certs podName:c84bbebc-7b6d-4e22-aacd-c98586717109 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:24.856421543 +0000 UTC m=+5.304130210 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c84bbebc-7b6d-4e22-aacd-c98586717109-metrics-certs") pod "network-metrics-daemon-flgwv" (UID: "c84bbebc-7b6d-4e22-aacd-c98586717109") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:33:22.856591 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:22.856515 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:33:22.856591 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:22.856553 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429722d3-91b1-4c92-b78e-19a12bc5bac9-original-pull-secret podName:429722d3-91b1-4c92-b78e-19a12bc5bac9 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:24.856542515 +0000 UTC m=+5.304251183 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/429722d3-91b1-4c92-b78e-19a12bc5bac9-original-pull-secret") pod "global-pull-secret-syncer-frkss" (UID: "429722d3-91b1-4c92-b78e-19a12bc5bac9") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:33:22.957020 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:22.956658 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxxl5\" (UniqueName: \"kubernetes.io/projected/17dae92a-ae2b-4c09-8001-48a81a8cf809-kube-api-access-mxxl5\") pod \"network-check-target-hh4r5\" (UID: \"17dae92a-ae2b-4c09-8001-48a81a8cf809\") " pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:22.957020 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:22.956872 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:33:22.957020 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:22.956892 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:33:22.957020 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:22.956906 2571 projected.go:196] Error preparing data for projected volume kube-api-access-mxxl5 for pod openshift-network-diagnostics/network-check-target-hh4r5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:33:22.957020 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:22.956966 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17dae92a-ae2b-4c09-8001-48a81a8cf809-kube-api-access-mxxl5 podName:17dae92a-ae2b-4c09-8001-48a81a8cf809 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:24.95694844 +0000 UTC m=+5.404657113 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mxxl5" (UniqueName: "kubernetes.io/projected/17dae92a-ae2b-4c09-8001-48a81a8cf809-kube-api-access-mxxl5") pod "network-check-target-hh4r5" (UID: "17dae92a-ae2b-4c09-8001-48a81a8cf809") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:33:23.071081 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:23.071043 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:23.071285 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:23.071209 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frkss" podUID="429722d3-91b1-4c92-b78e-19a12bc5bac9" Oct 30 18:33:23.071363 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:23.071298 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:23.071415 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:23.071369 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hh4r5" podUID="17dae92a-ae2b-4c09-8001-48a81a8cf809" Oct 30 18:33:23.071471 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:23.071434 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:23.071547 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:23.071527 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flgwv" podUID="c84bbebc-7b6d-4e22-aacd-c98586717109" Oct 30 18:33:23.084440 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:23.084363 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qnqxv" event={"ID":"c1ea54d3-410e-48cf-80f6-46be7218f433","Type":"ContainerStarted","Data":"825ca0cf96a8bbb51b048ef97bf527945b170c242b223a8d2f73c006a5fb0c9b"} Oct 30 18:33:23.086823 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:23.086797 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7c9gt" event={"ID":"02de5f04-abcb-440e-8e9e-55d41cd543e2","Type":"ContainerStarted","Data":"879e92e7e28f8539194240f70c7e058330b43243540df67b153118eeab2bda0f"} Oct 30 18:33:23.088886 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:23.088859 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f2745" event={"ID":"d676d57c-8007-406d-8b45-e3a6c33ac3c9","Type":"ContainerStarted","Data":"3df699b0fae271fb197e9b9b3b127585731af8cb1f09d00cb1a0b4368a49a505"} Oct 30 18:33:23.090416 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:23.090390 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-996pn" event={"ID":"3b5808a2-cbcf-4f2d-a401-191f429cc6be","Type":"ContainerStarted","Data":"3f796e66db196ea76ef51896a297e905fe8e8216b18fd5a23362c08c5f2df32f"} Oct 30 18:33:23.093119 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:23.093084 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" event={"ID":"e9a334f2-12b2-4188-8bfa-7bc69559f024","Type":"ContainerStarted","Data":"df73aee8aeec5eb2ccaf7e6e156a0cd993c9e672c36dd57025e1966f698b1637"} Oct 30 18:33:23.288719 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:23.288625 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:28:21 +0000 UTC" deadline="2028-04-10 21:17:45.396847529 +0000 UTC" Oct 30 18:33:23.288719 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:23.288670 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12674h44m22.108183053s" Oct 30 18:33:24.879011 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:24.878061 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84bbebc-7b6d-4e22-aacd-c98586717109-metrics-certs\") pod \"network-metrics-daemon-flgwv\" (UID: \"c84bbebc-7b6d-4e22-aacd-c98586717109\") " pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:24.879011 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:24.878181 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/429722d3-91b1-4c92-b78e-19a12bc5bac9-original-pull-secret\") pod \"global-pull-secret-syncer-frkss\" (UID: \"429722d3-91b1-4c92-b78e-19a12bc5bac9\") " pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:24.879011 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:24.878332 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:33:24.879011 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:24.878425 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429722d3-91b1-4c92-b78e-19a12bc5bac9-original-pull-secret podName:429722d3-91b1-4c92-b78e-19a12bc5bac9 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:28.878369115 +0000 UTC m=+9.326077792 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/429722d3-91b1-4c92-b78e-19a12bc5bac9-original-pull-secret") pod "global-pull-secret-syncer-frkss" (UID: "429722d3-91b1-4c92-b78e-19a12bc5bac9") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:33:24.879011 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:24.878502 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:33:24.879011 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:24.878542 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c84bbebc-7b6d-4e22-aacd-c98586717109-metrics-certs podName:c84bbebc-7b6d-4e22-aacd-c98586717109 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:28.878529795 +0000 UTC m=+9.326238477 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c84bbebc-7b6d-4e22-aacd-c98586717109-metrics-certs") pod "network-metrics-daemon-flgwv" (UID: "c84bbebc-7b6d-4e22-aacd-c98586717109") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:33:24.978608 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:24.978572 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxxl5\" (UniqueName: \"kubernetes.io/projected/17dae92a-ae2b-4c09-8001-48a81a8cf809-kube-api-access-mxxl5\") pod \"network-check-target-hh4r5\" (UID: \"17dae92a-ae2b-4c09-8001-48a81a8cf809\") " pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:24.978793 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:24.978755 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:33:24.978793 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:24.978773 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:33:24.978793 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:24.978788 2571 projected.go:196] Error preparing data for projected volume kube-api-access-mxxl5 for pod openshift-network-diagnostics/network-check-target-hh4r5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:33:24.978947 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:24.978847 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17dae92a-ae2b-4c09-8001-48a81a8cf809-kube-api-access-mxxl5 podName:17dae92a-ae2b-4c09-8001-48a81a8cf809 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:28.978828239 +0000 UTC m=+9.426536916 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mxxl5" (UniqueName: "kubernetes.io/projected/17dae92a-ae2b-4c09-8001-48a81a8cf809-kube-api-access-mxxl5") pod "network-check-target-hh4r5" (UID: "17dae92a-ae2b-4c09-8001-48a81a8cf809") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:33:25.071402 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:25.071364 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:25.071558 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:25.071510 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flgwv" podUID="c84bbebc-7b6d-4e22-aacd-c98586717109" Oct 30 18:33:25.071622 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:25.071591 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:25.071702 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:25.071657 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frkss" podUID="429722d3-91b1-4c92-b78e-19a12bc5bac9" Oct 30 18:33:25.071742 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:25.071709 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:25.071809 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:25.071770 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hh4r5" podUID="17dae92a-ae2b-4c09-8001-48a81a8cf809" Oct 30 18:33:26.113184 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:26.112521 2571 generic.go:356] "Generic (PLEG): container finished" podID="76ae57dd60561f1d7425076e8b67e7e3" containerID="cd94820c78f9f788b6873f16f8768bc671b5eb566a22de9793ca9f0c13baa86e" exitCode=0 Oct 30 18:33:26.113184 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:26.112626 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal" event={"ID":"76ae57dd60561f1d7425076e8b67e7e3","Type":"ContainerDied","Data":"cd94820c78f9f788b6873f16f8768bc671b5eb566a22de9793ca9f0c13baa86e"} Oct 30 18:33:26.132468 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:26.132402 2571 generic.go:356] "Generic (PLEG): container finished" podID="cd3cee01-8466-456c-9b3d-c4f1fc5b4a89" containerID="0fde78119496ac1fc1a8ae3a24c28da592c05ada6f5002fe15da0ec12cb1b820" exitCode=0 Oct 30 18:33:26.132635 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:26.132579 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxpzg" event={"ID":"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89","Type":"ContainerDied","Data":"0fde78119496ac1fc1a8ae3a24c28da592c05ada6f5002fe15da0ec12cb1b820"} Oct 30 18:33:26.141407 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:26.141307 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sd88x" event={"ID":"002053ca-649b-4a56-9ffd-25406d82fe99","Type":"ContainerStarted","Data":"c919adfee2ca61676531c4958e0a70d643daa63489fba69cce09b44adacc9833"} Oct 30 18:33:26.339571 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:26.338940 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sd88x" podStartSLOduration=2.173526275 podStartE2EDuration="6.338918107s" totalImagesPullingTime="4.165391832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:26.338186937 +0000 UTC m=+6.785895623" watchObservedRunningTime="2026-10-30 18:33:26.338918107 +0000 UTC m=+6.786626793" Oct 30 18:33:27.078892 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:27.076471 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:27.078892 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:27.076641 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hh4r5" podUID="17dae92a-ae2b-4c09-8001-48a81a8cf809" Oct 30 18:33:27.078892 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:27.076725 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:27.078892 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:27.076852 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flgwv" podUID="c84bbebc-7b6d-4e22-aacd-c98586717109" Oct 30 18:33:27.078892 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:27.076913 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:27.078892 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:27.077006 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frkss" podUID="429722d3-91b1-4c92-b78e-19a12bc5bac9" Oct 30 18:33:27.147545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:27.146791 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal" event={"ID":"76ae57dd60561f1d7425076e8b67e7e3","Type":"ContainerStarted","Data":"dd0b66cf63f343b74fb38ce92507c5bc6bb98b5e94d3f3ef4a991378c7be7e70"} Oct 30 18:33:27.150900 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:27.150830 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal" event={"ID":"d156aeb8e5a3750f522b65c4725c3f82","Type":"ContainerStarted","Data":"8517856f84378358cb894616c72fb46b303033f0309ca4c918656896e1032f03"} Oct 30 18:33:27.266927 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:27.266190 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal" podStartSLOduration=7.266170472 podStartE2EDuration="7.266170472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:27.265233534 +0000 UTC m=+7.712942220" watchObservedRunningTime="2026-10-30 18:33:27.266170472 +0000 UTC m=+7.713879159" Oct 30 18:33:27.381734 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:27.381625 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal" podStartSLOduration=7.3816073509999995 podStartE2EDuration="7.381607351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:27.381397324 +0000 UTC m=+7.829106011" watchObservedRunningTime="2026-10-30 18:33:27.381607351 +0000 UTC m=+7.829316043" Oct 30 18:33:28.913660 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:28.913231 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/429722d3-91b1-4c92-b78e-19a12bc5bac9-original-pull-secret\") pod \"global-pull-secret-syncer-frkss\" (UID: \"429722d3-91b1-4c92-b78e-19a12bc5bac9\") " pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:28.913660 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:28.913278 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84bbebc-7b6d-4e22-aacd-c98586717109-metrics-certs\") pod \"network-metrics-daemon-flgwv\" (UID: \"c84bbebc-7b6d-4e22-aacd-c98586717109\") " pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:28.913660 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:28.913427 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:33:28.913660 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:28.913482 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c84bbebc-7b6d-4e22-aacd-c98586717109-metrics-certs podName:c84bbebc-7b6d-4e22-aacd-c98586717109 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:36.913470077 +0000 UTC m=+17.361178753 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c84bbebc-7b6d-4e22-aacd-c98586717109-metrics-certs") pod "network-metrics-daemon-flgwv" (UID: "c84bbebc-7b6d-4e22-aacd-c98586717109") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:33:28.913660 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:28.913544 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:33:28.913660 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:28.913578 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429722d3-91b1-4c92-b78e-19a12bc5bac9-original-pull-secret podName:429722d3-91b1-4c92-b78e-19a12bc5bac9 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:36.913565283 +0000 UTC m=+17.361273960 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/429722d3-91b1-4c92-b78e-19a12bc5bac9-original-pull-secret") pod "global-pull-secret-syncer-frkss" (UID: "429722d3-91b1-4c92-b78e-19a12bc5bac9") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:33:28.925918 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:28.925163 2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sd88x" Oct 30 18:33:28.926578 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:28.926348 2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sd88x" Oct 30 18:33:29.014589 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:29.013745 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxxl5\" (UniqueName: \"kubernetes.io/projected/17dae92a-ae2b-4c09-8001-48a81a8cf809-kube-api-access-mxxl5\") pod \"network-check-target-hh4r5\" (UID: \"17dae92a-ae2b-4c09-8001-48a81a8cf809\") " pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:29.014589 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:29.014292 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:33:29.014589 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:29.014317 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:33:29.014589 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:29.014331 2571 projected.go:196] Error preparing data for projected volume kube-api-access-mxxl5 for pod openshift-network-diagnostics/network-check-target-hh4r5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:33:29.014589 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:29.014385 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17dae92a-ae2b-4c09-8001-48a81a8cf809-kube-api-access-mxxl5 podName:17dae92a-ae2b-4c09-8001-48a81a8cf809 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:37.014366382 +0000 UTC m=+17.462075059 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mxxl5" (UniqueName: "kubernetes.io/projected/17dae92a-ae2b-4c09-8001-48a81a8cf809-kube-api-access-mxxl5") pod "network-check-target-hh4r5" (UID: "17dae92a-ae2b-4c09-8001-48a81a8cf809") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:33:29.070989 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:29.070523 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:29.070989 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:29.070650 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hh4r5" podUID="17dae92a-ae2b-4c09-8001-48a81a8cf809" Oct 30 18:33:29.070989 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:29.070719 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:29.070989 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:29.070805 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flgwv" podUID="c84bbebc-7b6d-4e22-aacd-c98586717109" Oct 30 18:33:29.070989 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:29.070854 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:29.070989 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:29.070909 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frkss" podUID="429722d3-91b1-4c92-b78e-19a12bc5bac9" Oct 30 18:33:29.163262 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:29.163092 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" event={"ID":"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096","Type":"ContainerStarted","Data":"8ee45dc0a05a87bb5d409916769dcbdc9545005bf31370cb0a8a41479029615c"} Oct 30 18:33:29.168261 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:29.168073 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7c9gt" event={"ID":"02de5f04-abcb-440e-8e9e-55d41cd543e2","Type":"ContainerStarted","Data":"a46f4263ec5b4334dfc6e75be9ad099471db880c10097e4930e96bf21ad49e78"} Oct 30 18:33:30.831452 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:30.831421 2571 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 18:33:31.011318 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:31.011190 2571 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:33:30.831447479Z","UUID":"5a976af5-df7c-45c8-abd2-e71e425cff58","Handler":null,"Name":"","Endpoint":""} Oct 30 18:33:31.013264 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:31.013232 2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 18:33:31.013264 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:31.013272 2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 18:33:31.071360 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:31.071321 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:31.071527 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:31.071464 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frkss" podUID="429722d3-91b1-4c92-b78e-19a12bc5bac9" Oct 30 18:33:31.071585 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:31.071539 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:31.071637 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:31.071605 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hh4r5" podUID="17dae92a-ae2b-4c09-8001-48a81a8cf809" Oct 30 18:33:31.071689 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:31.071669 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:31.071769 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:31.071755 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flgwv" podUID="c84bbebc-7b6d-4e22-aacd-c98586717109" Oct 30 18:33:31.177066 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:31.177030 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7jq9m" event={"ID":"2d258432-7e24-4b20-a0e8-258aa1e1a4a5","Type":"ContainerStarted","Data":"b4ae364e6cb325fc8004f5732d969911e89b686dd0edc83ad922d3793c99ac28"} Oct 30 18:33:31.182789 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:31.182690 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" event={"ID":"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096","Type":"ContainerStarted","Data":"18321af0017c7ea6b358d509c6186d47e1a852ccbd8d948357481ffb5648c929"} Oct 30 18:33:31.188803 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:31.188719 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f2745" event={"ID":"d676d57c-8007-406d-8b45-e3a6c33ac3c9","Type":"ContainerStarted","Data":"2757c0959173da9b66cb07b80162e266f2884ca7bb2c023d1e595f4fab5f14eb"} Oct 30 18:33:31.294097 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:31.294043 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-7jq9m" podStartSLOduration=2.651210152 podStartE2EDuration="11.294030556s" totalImagesPullingTime="8.642820404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:33:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:31.293874748 +0000 UTC m=+11.741583434" watchObservedRunningTime="2026-10-30 18:33:31.294030556 +0000 UTC m=+11.741739242" Oct 30 18:33:31.294500 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:31.294469 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7c9gt" podStartSLOduration=5.40824669 podStartE2EDuration="11.294458556s" totalImagesPullingTime="5.886211866s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:33:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:29.283785479 +0000 UTC m=+9.731494163" watchObservedRunningTime="2026-10-30 18:33:31.294458556 +0000 UTC m=+11.742167240" Oct 30 18:33:31.412265 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:31.412194 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-f2745" podStartSLOduration=2.754417848 podStartE2EDuration="11.412170598s" totalImagesPullingTime="8.65775275s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:33:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:31.411221348 +0000 UTC m=+11.858930034" watchObservedRunningTime="2026-10-30 18:33:31.412170598 +0000 UTC m=+11.859879283" Oct 30 18:33:31.495956 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:31.495921 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sd88x" Oct 30 18:33:31.496123 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:31.496078 2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 18:33:31.497005 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:31.496986 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sd88x" Oct 30 18:33:32.196624 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:32.196341 2571 generic.go:356] "Generic (PLEG): container finished" podID="cd3cee01-8466-456c-9b3d-c4f1fc5b4a89" containerID="f978acb9238f8f2c5ccf020e519f220cab738b8c8946bfc396622cd62e1ceee5" exitCode=0 Oct 30 18:33:32.197774 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:32.197743 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxpzg" event={"ID":"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89","Type":"ContainerDied","Data":"f978acb9238f8f2c5ccf020e519f220cab738b8c8946bfc396622cd62e1ceee5"} Oct 30 18:33:33.070625 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:33.070601 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:33.070705 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:33.070635 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:33.070750 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:33.070723 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frkss" podUID="429722d3-91b1-4c92-b78e-19a12bc5bac9" Oct 30 18:33:33.070840 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:33.070815 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:33.070946 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:33.070916 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flgwv" podUID="c84bbebc-7b6d-4e22-aacd-c98586717109" Oct 30 18:33:33.071002 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:33.070965 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hh4r5" podUID="17dae92a-ae2b-4c09-8001-48a81a8cf809" Oct 30 18:33:33.201254 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:33.201216 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qnqxv" event={"ID":"c1ea54d3-410e-48cf-80f6-46be7218f433","Type":"ContainerStarted","Data":"a0c3a28e36ba4c0ce8abf134190a23c83876ed9c505b8f985997b11c5fb71cf1"} Oct 30 18:33:33.204724 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:33.204686 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" event={"ID":"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096","Type":"ContainerStarted","Data":"b14b532c4a8a529b1c1161ea10973c5403b002a22945e62e9961cc7a77d9069d"} Oct 30 18:33:33.207076 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:33.207039 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-996pn" event={"ID":"3b5808a2-cbcf-4f2d-a401-191f429cc6be","Type":"ContainerStarted","Data":"27a42e2c30bd65aa86ff4db6f2f765dddafa393de04797794a7af881aa96af54"} Oct 30 18:33:33.317981 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:33.317918 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qnqxv" podStartSLOduration=4.627649397 podStartE2EDuration="13.317898668s" totalImagesPullingTime="8.690249271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:33:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:33.317635291 +0000 UTC m=+13.765343965" watchObservedRunningTime="2026-10-30 18:33:33.317898668 +0000 UTC m=+13.765607357" Oct 30 18:33:33.436214 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:33.436104 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" podStartSLOduration=3.246963103 podStartE2EDuration="13.436090381s" totalImagesPullingTime="10.189127278s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:33:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:33.436016534 +0000 UTC m=+13.883725217" watchObservedRunningTime="2026-10-30 18:33:33.436090381 +0000 UTC m=+13.883799062" Oct 30 18:33:33.553938 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:33.553885 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-996pn" podStartSLOduration=2.854259694 podStartE2EDuration="13.553869861s" totalImagesPullingTime="10.699610167s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:33:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:33.553617211 +0000 UTC m=+14.001325906" watchObservedRunningTime="2026-10-30 18:33:33.553869861 +0000 UTC m=+14.001578546" Oct 30 18:33:34.211545 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:34.211508 2571 generic.go:356] "Generic (PLEG): container finished" podID="cd3cee01-8466-456c-9b3d-c4f1fc5b4a89" containerID="ed2d1d491dab8914fa34ddb3a832936ad46b658ea1fbaaf6952c9188a1ec3441" exitCode=0 Oct 30 18:33:34.212162 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:34.211619 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxpzg" event={"ID":"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89","Type":"ContainerDied","Data":"ed2d1d491dab8914fa34ddb3a832936ad46b658ea1fbaaf6952c9188a1ec3441"} Oct 30 18:33:35.071020 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:35.070979 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:35.071249 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:35.071147 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flgwv" podUID="c84bbebc-7b6d-4e22-aacd-c98586717109" Oct 30 18:33:35.071604 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:35.071579 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:35.071677 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:35.071666 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frkss" podUID="429722d3-91b1-4c92-b78e-19a12bc5bac9" Oct 30 18:33:35.071742 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:35.071725 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:35.071805 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:35.071786 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hh4r5" podUID="17dae92a-ae2b-4c09-8001-48a81a8cf809" Oct 30 18:33:36.218320 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:36.218281 2571 generic.go:356] "Generic (PLEG): container finished" podID="cd3cee01-8466-456c-9b3d-c4f1fc5b4a89" containerID="872122e5c58ec51c2cf2cf24b2665a6fffb802db24ceef797e1b084a48b4ef42" exitCode=0 Oct 30 18:33:36.219004 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:36.218359 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxpzg" event={"ID":"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89","Type":"ContainerDied","Data":"872122e5c58ec51c2cf2cf24b2665a6fffb802db24ceef797e1b084a48b4ef42"} Oct 30 18:33:36.979361 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:36.979317 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84bbebc-7b6d-4e22-aacd-c98586717109-metrics-certs\") pod \"network-metrics-daemon-flgwv\" (UID: \"c84bbebc-7b6d-4e22-aacd-c98586717109\") " pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:36.979554 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:36.979411 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/429722d3-91b1-4c92-b78e-19a12bc5bac9-original-pull-secret\") pod \"global-pull-secret-syncer-frkss\" (UID: \"429722d3-91b1-4c92-b78e-19a12bc5bac9\") " pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:36.979554 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:36.979493 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:33:36.979554 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:36.979522 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:33:36.979732 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:36.979579 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c84bbebc-7b6d-4e22-aacd-c98586717109-metrics-certs podName:c84bbebc-7b6d-4e22-aacd-c98586717109 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:52.979556987 +0000 UTC m=+33.427265651 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c84bbebc-7b6d-4e22-aacd-c98586717109-metrics-certs") pod "network-metrics-daemon-flgwv" (UID: "c84bbebc-7b6d-4e22-aacd-c98586717109") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:33:36.979732 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:36.979601 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429722d3-91b1-4c92-b78e-19a12bc5bac9-original-pull-secret podName:429722d3-91b1-4c92-b78e-19a12bc5bac9 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:52.979590097 +0000 UTC m=+33.427298765 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/429722d3-91b1-4c92-b78e-19a12bc5bac9-original-pull-secret") pod "global-pull-secret-syncer-frkss" (UID: "429722d3-91b1-4c92-b78e-19a12bc5bac9") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:33:37.070505 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:37.070336 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:37.070698 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:37.070378 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:37.070698 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:37.070589 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frkss" podUID="429722d3-91b1-4c92-b78e-19a12bc5bac9" Oct 30 18:33:37.070698 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:37.070666 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hh4r5" podUID="17dae92a-ae2b-4c09-8001-48a81a8cf809" Oct 30 18:33:37.070698 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:37.070384 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:37.070834 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:37.070732 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flgwv" podUID="c84bbebc-7b6d-4e22-aacd-c98586717109" Oct 30 18:33:37.080178 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:37.080150 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxxl5\" (UniqueName: \"kubernetes.io/projected/17dae92a-ae2b-4c09-8001-48a81a8cf809-kube-api-access-mxxl5\") pod \"network-check-target-hh4r5\" (UID: \"17dae92a-ae2b-4c09-8001-48a81a8cf809\") " pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:37.080309 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:37.080296 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:33:37.080348 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:37.080312 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:33:37.080348 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:37.080322 2571 projected.go:196] Error preparing data for projected volume kube-api-access-mxxl5 for pod openshift-network-diagnostics/network-check-target-hh4r5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:33:37.080420 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:37.080384 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/17dae92a-ae2b-4c09-8001-48a81a8cf809-kube-api-access-mxxl5 podName:17dae92a-ae2b-4c09-8001-48a81a8cf809 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:53.08036441 +0000 UTC m=+33.528073088 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mxxl5" (UniqueName: "kubernetes.io/projected/17dae92a-ae2b-4c09-8001-48a81a8cf809-kube-api-access-mxxl5") pod "network-check-target-hh4r5" (UID: "17dae92a-ae2b-4c09-8001-48a81a8cf809") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:33:37.222406 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:37.222371 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" event={"ID":"e9a334f2-12b2-4188-8bfa-7bc69559f024","Type":"ContainerStarted","Data":"ceaf0b68d88ec7f095b489056824471e6729d4a2138a993ca8503ec650220158"} Oct 30 18:33:37.222805 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:37.222472 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" event={"ID":"e9a334f2-12b2-4188-8bfa-7bc69559f024","Type":"ContainerStarted","Data":"224121d83f91ab9e8a03fc9a8b4a9754f2582fe9a1aa2e45096a856da12d8f19"} Oct 30 18:33:37.222805 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:37.222502 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" event={"ID":"e9a334f2-12b2-4188-8bfa-7bc69559f024","Type":"ContainerStarted","Data":"f9dcfe0bea9f61e35fcde2bacb2895961827055b368c6df0cbbaa2d9ab189f19"} Oct 30 18:33:38.229191 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:38.228934 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" event={"ID":"e9a334f2-12b2-4188-8bfa-7bc69559f024","Type":"ContainerStarted","Data":"aedd34592baf3c26c6858195eada2ebb76839e14be8be8b9af185a8df562a5d6"} Oct 30 18:33:38.229724 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:38.229252 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" event={"ID":"e9a334f2-12b2-4188-8bfa-7bc69559f024","Type":"ContainerStarted","Data":"26cc3bbdbd41ead1acd171d7230750315574d63f2fd832d111fa9d15782bd95f"} Oct 30 18:33:38.229724 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:38.229304 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" event={"ID":"e9a334f2-12b2-4188-8bfa-7bc69559f024","Type":"ContainerStarted","Data":"98f1de16a6c435a84c6248da8f1435f2d47e0222b3cff5b32f6e1fed2a9ef802"} Oct 30 18:33:39.070965 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:39.070936 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:39.070965 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:39.070957 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:39.071254 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:39.070950 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:39.071254 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:39.071044 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flgwv" podUID="c84bbebc-7b6d-4e22-aacd-c98586717109" Oct 30 18:33:39.071254 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:39.071118 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hh4r5" podUID="17dae92a-ae2b-4c09-8001-48a81a8cf809" Oct 30 18:33:39.071254 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:39.071206 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frkss" podUID="429722d3-91b1-4c92-b78e-19a12bc5bac9" Oct 30 18:33:39.234261 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:39.234216 2571 generic.go:356] "Generic (PLEG): container finished" podID="cd3cee01-8466-456c-9b3d-c4f1fc5b4a89" containerID="349c62804450a51f6b8f9cc0a4dfb14b2380dc295110fac64b77647e654b416f" exitCode=0 Oct 30 18:33:39.234261 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:39.234259 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxpzg" event={"ID":"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89","Type":"ContainerDied","Data":"349c62804450a51f6b8f9cc0a4dfb14b2380dc295110fac64b77647e654b416f"} Oct 30 18:33:40.239749 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:40.239710 2571 generic.go:356] "Generic (PLEG): container finished" podID="cd3cee01-8466-456c-9b3d-c4f1fc5b4a89" containerID="d53eb76f3e3962af7b50fb95d4ba5128406718e08e8626b20585e972a1a61fec" exitCode=0 Oct 30 18:33:40.240429 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:40.239765 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxpzg" event={"ID":"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89","Type":"ContainerDied","Data":"d53eb76f3e3962af7b50fb95d4ba5128406718e08e8626b20585e972a1a61fec"} Oct 30 18:33:40.243112 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:40.243087 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" event={"ID":"e9a334f2-12b2-4188-8bfa-7bc69559f024","Type":"ContainerStarted","Data":"ba178a0d3578a5deb6a5a551416ebc1132d194607302191266fd6cc62362103a"} Oct 30 18:33:41.070436 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:41.070397 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:41.070606 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:41.070444 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:41.070606 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:41.070406 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:41.070606 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:41.070520 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frkss" podUID="429722d3-91b1-4c92-b78e-19a12bc5bac9" Oct 30 18:33:41.070606 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:41.070571 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hh4r5" podUID="17dae92a-ae2b-4c09-8001-48a81a8cf809" Oct 30 18:33:41.070730 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:41.070655 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flgwv" podUID="c84bbebc-7b6d-4e22-aacd-c98586717109" Oct 30 18:33:41.248481 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:41.248454 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxpzg" event={"ID":"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89","Type":"ContainerStarted","Data":"ec9521db7dd03b8371d8e9795a573b90b427ac10d68c3471b5e73541460d0b5c"} Oct 30 18:33:41.420492 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:41.420386 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nxpzg" podStartSLOduration=8.861465593 podStartE2EDuration="21.420372817s" totalImagesPullingTime="12.558907224s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:33:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:41.420189924 +0000 UTC m=+21.867898606" watchObservedRunningTime="2026-10-30 18:33:41.420372817 +0000 UTC m=+21.868081501" Oct 30 18:33:42.254805 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:42.254562 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" event={"ID":"e9a334f2-12b2-4188-8bfa-7bc69559f024","Type":"ContainerStarted","Data":"a30fe5fab0962dbb67b09f7091a18553950d56fa16a021a1794868dd401fdcd1"} Oct 30 18:33:42.539449 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:42.539352 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" podStartSLOduration=7.865961459 podStartE2EDuration="22.539335639s" totalImagesPullingTime="14.67337418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:33:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:42.53894941 +0000 UTC m=+22.986658095" watchObservedRunningTime="2026-10-30 18:33:42.539335639 +0000 UTC m=+22.987044324" Oct 30 18:33:43.071439 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:43.071402 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:43.071439 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:43.071433 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:43.071649 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:43.071525 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flgwv" podUID="c84bbebc-7b6d-4e22-aacd-c98586717109" Oct 30 18:33:43.071649 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:43.071568 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:43.071711 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:43.071652 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frkss" podUID="429722d3-91b1-4c92-b78e-19a12bc5bac9" Oct 30 18:33:43.071746 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:43.071709 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hh4r5" podUID="17dae92a-ae2b-4c09-8001-48a81a8cf809" Oct 30 18:33:43.257449 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:43.257420 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:43.257897 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:43.257506 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:43.257897 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:43.257542 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:43.274186 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:43.274159 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:43.278653 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:43.278632 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:33:44.465464 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:44.465378 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hh4r5"] Oct 30 18:33:44.466003 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:44.465519 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:44.466003 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:44.465626 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hh4r5" podUID="17dae92a-ae2b-4c09-8001-48a81a8cf809" Oct 30 18:33:44.466182 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:44.466158 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-flgwv"] Oct 30 18:33:44.466375 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:44.466347 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:44.466467 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:44.466437 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flgwv" podUID="c84bbebc-7b6d-4e22-aacd-c98586717109" Oct 30 18:33:44.468292 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:44.468266 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-frkss"] Oct 30 18:33:44.468385 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:44.468355 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:44.468460 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:44.468439 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frkss" podUID="429722d3-91b1-4c92-b78e-19a12bc5bac9" Oct 30 18:33:46.071390 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:46.071356 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:46.071762 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:46.071356 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:46.071762 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:46.071468 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frkss" podUID="429722d3-91b1-4c92-b78e-19a12bc5bac9" Oct 30 18:33:46.071762 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:46.071565 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hh4r5" podUID="17dae92a-ae2b-4c09-8001-48a81a8cf809" Oct 30 18:33:46.071762 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:46.071356 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:46.071762 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:46.071684 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flgwv" podUID="c84bbebc-7b6d-4e22-aacd-c98586717109" Oct 30 18:33:48.071414 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:48.071378 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:48.071929 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:48.071385 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:48.071929 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:48.071507 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hh4r5" podUID="17dae92a-ae2b-4c09-8001-48a81a8cf809" Oct 30 18:33:48.071929 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:48.071522 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:48.071929 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:48.071614 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flgwv" podUID="c84bbebc-7b6d-4e22-aacd-c98586717109" Oct 30 18:33:48.071929 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:48.071680 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-frkss" podUID="429722d3-91b1-4c92-b78e-19a12bc5bac9" Oct 30 18:33:49.847912 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:49.847879 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeReady" Oct 30 18:33:49.848327 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:49.848016 2571 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 18:33:49.964659 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:49.964564 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hjrzq"] Oct 30 18:33:49.977219 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:49.977181 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-49jb4"] Oct 30 18:33:49.977393 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:49.977325 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hjrzq" Oct 30 18:33:49.991226 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:49.991194 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-49jb4" Oct 30 18:33:50.031264 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.031231 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 18:33:50.031437 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.031267 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 18:33:50.031601 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.031585 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 18:33:50.031688 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.031638 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-sqz88\"" Oct 30 18:33:50.049332 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.049255 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hjrzq"] Oct 30 18:33:50.050785 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.050279 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-49jb4"] Oct 30 18:33:50.071334 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.071295 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:50.071334 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.071322 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:50.071334 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.071335 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:50.072743 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.072711 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:33:50.072872 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.072748 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-tmp-dir\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:33:50.072872 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.072773 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/886a6628-cf58-4784-85e0-d1ca14a2403c-cert\") pod \"ingress-canary-hjrzq\" (UID: \"886a6628-cf58-4784-85e0-d1ca14a2403c\") " pod="openshift-ingress-canary/ingress-canary-hjrzq" Oct 30 18:33:50.073016 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.072938 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc7ct\" (UniqueName: \"kubernetes.io/projected/886a6628-cf58-4784-85e0-d1ca14a2403c-kube-api-access-xc7ct\") pod \"ingress-canary-hjrzq\" (UID: \"886a6628-cf58-4784-85e0-d1ca14a2403c\") " pod="openshift-ingress-canary/ingress-canary-hjrzq" Oct 30 18:33:50.073016 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.072972 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-config-volume\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:33:50.073160 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.073067 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpjjc\" (UniqueName: \"kubernetes.io/projected/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-kube-api-access-wpjjc\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:33:50.133373 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.133334 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 18:33:50.133373 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.133351 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 18:33:50.133640 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.133418 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 18:33:50.133640 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.133461 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 18:33:50.133640 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.133574 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-8wmd6\"" Oct 30 18:33:50.133768 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.133652 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wkpbb\"" Oct 30 18:33:50.133768 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.133687 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 18:33:50.133839 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.133818 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lmrrc\"" Oct 30 18:33:50.133992 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.133980 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 18:33:50.134068 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.134055 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-727z7\"" Oct 30 18:33:50.174003 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.173967 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-config-volume\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:33:50.174191 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.174038 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpjjc\" (UniqueName: \"kubernetes.io/projected/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-kube-api-access-wpjjc\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:33:50.174191 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.174086 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:33:50.174191 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.174107 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-tmp-dir\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:33:50.174191 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.174152 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/886a6628-cf58-4784-85e0-d1ca14a2403c-cert\") pod \"ingress-canary-hjrzq\" (UID: \"886a6628-cf58-4784-85e0-d1ca14a2403c\") " pod="openshift-ingress-canary/ingress-canary-hjrzq" Oct 30 18:33:50.174409 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.174208 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc7ct\" (UniqueName: \"kubernetes.io/projected/886a6628-cf58-4784-85e0-d1ca14a2403c-kube-api-access-xc7ct\") pod \"ingress-canary-hjrzq\" (UID: \"886a6628-cf58-4784-85e0-d1ca14a2403c\") " pod="openshift-ingress-canary/ingress-canary-hjrzq" Oct 30 18:33:50.174409 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:50.174218 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 18:33:50.174537 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:50.174513 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls podName:eaa9cd18-eea8-4b24-b6af-c8cd6e278011 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:50.674487913 +0000 UTC m=+31.122196592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls") pod "dns-default-49jb4" (UID: "eaa9cd18-eea8-4b24-b6af-c8cd6e278011") : secret "dns-default-metrics-tls" not found Oct 30 18:33:50.174730 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.174708 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-tmp-dir\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:33:50.174945 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.174931 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-config-volume\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:33:50.178378 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.178244 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/886a6628-cf58-4784-85e0-d1ca14a2403c-cert\") pod \"ingress-canary-hjrzq\" (UID: \"886a6628-cf58-4784-85e0-d1ca14a2403c\") " pod="openshift-ingress-canary/ingress-canary-hjrzq" Oct 30 18:33:50.236620 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.236547 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc7ct\" (UniqueName: \"kubernetes.io/projected/886a6628-cf58-4784-85e0-d1ca14a2403c-kube-api-access-xc7ct\") pod \"ingress-canary-hjrzq\" (UID: \"886a6628-cf58-4784-85e0-d1ca14a2403c\") " pod="openshift-ingress-canary/ingress-canary-hjrzq" Oct 30 18:33:50.236620 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.236604 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpjjc\" (UniqueName: \"kubernetes.io/projected/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-kube-api-access-wpjjc\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:33:50.301336 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.301294 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hjrzq" Oct 30 18:33:50.676567 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.676521 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:33:50.676727 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:50.676691 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 18:33:50.676790 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:50.676778 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls podName:eaa9cd18-eea8-4b24-b6af-c8cd6e278011 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:51.676760436 +0000 UTC m=+32.124469129 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls") pod "dns-default-49jb4" (UID: "eaa9cd18-eea8-4b24-b6af-c8cd6e278011") : secret "dns-default-metrics-tls" not found Oct 30 18:33:50.706942 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:50.706912 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hjrzq"] Oct 30 18:33:50.711024 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:50.710992 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod886a6628_cf58_4784_85e0_d1ca14a2403c.slice/crio-ee924fccdf6e416545bca8e22fe329ed989ffea1a1bd6e08a78736d3da4fa863 WatchSource:0}: Error finding container ee924fccdf6e416545bca8e22fe329ed989ffea1a1bd6e08a78736d3da4fa863: Status 404 returned error can't find the container with id ee924fccdf6e416545bca8e22fe329ed989ffea1a1bd6e08a78736d3da4fa863 Oct 30 18:33:51.277777 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:51.277736 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hjrzq" event={"ID":"886a6628-cf58-4784-85e0-d1ca14a2403c","Type":"ContainerStarted","Data":"ee924fccdf6e416545bca8e22fe329ed989ffea1a1bd6e08a78736d3da4fa863"} Oct 30 18:33:51.683656 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:51.683562 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:33:51.683801 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:51.683763 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 18:33:51.683849 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:51.683838 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls podName:eaa9cd18-eea8-4b24-b6af-c8cd6e278011 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:53.683822574 +0000 UTC m=+34.131531237 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls") pod "dns-default-49jb4" (UID: "eaa9cd18-eea8-4b24-b6af-c8cd6e278011") : secret "dns-default-metrics-tls" not found Oct 30 18:33:51.905228 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:51.905194 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-v9jf7"] Oct 30 18:33:51.907037 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:51.907017 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:33:51.972183 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:51.972086 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v9jf7"] Oct 30 18:33:51.986808 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:51.986767 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:33:51.986962 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:51.986826 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/af0639e4-bb41-4099-a9cf-03678c968c0b-data-volume\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:33:51.987002 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:51.986977 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af0639e4-bb41-4099-a9cf-03678c968c0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:33:51.987077 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:51.987053 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/af0639e4-bb41-4099-a9cf-03678c968c0b-crio-socket\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:33:51.987192 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:51.987102 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rfpj\" (UniqueName: \"kubernetes.io/projected/af0639e4-bb41-4099-a9cf-03678c968c0b-kube-api-access-9rfpj\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:33:52.013549 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.013512 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 18:33:52.013736 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.013612 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k76xp\"" Oct 30 18:33:52.016220 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.016192 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 18:33:52.016382 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.016242 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 18:33:52.016382 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.016243 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 18:33:52.088188 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.088145 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/af0639e4-bb41-4099-a9cf-03678c968c0b-data-volume\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:33:52.088380 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.088201 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af0639e4-bb41-4099-a9cf-03678c968c0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:33:52.088380 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.088257 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/af0639e4-bb41-4099-a9cf-03678c968c0b-crio-socket\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:33:52.088380 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.088288 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rfpj\" (UniqueName: \"kubernetes.io/projected/af0639e4-bb41-4099-a9cf-03678c968c0b-kube-api-access-9rfpj\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:33:52.088487 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.088382 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:33:52.088526 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.088484 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/af0639e4-bb41-4099-a9cf-03678c968c0b-data-volume\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:33:52.088526 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:52.088511 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 18:33:52.088581 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:52.088562 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls podName:af0639e4-bb41-4099-a9cf-03678c968c0b nodeName:}" failed. No retries permitted until 2026-10-30 18:33:52.588545958 +0000 UTC m=+33.036254621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9jf7" (UID: "af0639e4-bb41-4099-a9cf-03678c968c0b") : secret "insights-runtime-extractor-tls" not found Oct 30 18:33:52.088696 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.088668 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/af0639e4-bb41-4099-a9cf-03678c968c0b-crio-socket\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:33:52.088828 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:52.088740 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 18:33:52.088828 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:52.088800 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af0639e4-bb41-4099-a9cf-03678c968c0b-kube-rbac-proxy-cm podName:af0639e4-bb41-4099-a9cf-03678c968c0b nodeName:}" failed. No retries permitted until 2026-10-30 18:33:52.588782827 +0000 UTC m=+33.036491506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/af0639e4-bb41-4099-a9cf-03678c968c0b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v9jf7" (UID: "af0639e4-bb41-4099-a9cf-03678c968c0b") : configmap "kube-rbac-proxy" not found Oct 30 18:33:52.104662 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.104627 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-zhmhr-2v9dv"] Oct 30 18:33:52.106754 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.106727 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" Oct 30 18:33:52.150559 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.150530 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rfpj\" (UniqueName: \"kubernetes.io/projected/af0639e4-bb41-4099-a9cf-03678c968c0b-kube-api-access-9rfpj\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:33:52.159903 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.159871 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 18:33:52.159903 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.159898 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lwjtg\"" Oct 30 18:33:52.188908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.188872 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-service-ca-bundle\") pod \"periodic-gathering-zhmhr-2v9dv\" (UID: \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\") " pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" Oct 30 18:33:52.188908 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.188908 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-trusted-ca-bundle\") pod \"periodic-gathering-zhmhr-2v9dv\" (UID: \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\") " pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" Oct 30 18:33:52.189158 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.189020 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-archives-path\") pod \"periodic-gathering-zhmhr-2v9dv\" (UID: \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\") " pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" Oct 30 18:33:52.189158 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.189059 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rswb\" (UniqueName: \"kubernetes.io/projected/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-kube-api-access-5rswb\") pod \"periodic-gathering-zhmhr-2v9dv\" (UID: \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\") " pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" Oct 30 18:33:52.210849 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.210816 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 18:33:52.211917 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.211893 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-zhmhr-2v9dv"] Oct 30 18:33:52.290073 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.290033 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-service-ca-bundle\") pod \"periodic-gathering-zhmhr-2v9dv\" (UID: \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\") " pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" Oct 30 18:33:52.290073 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.290074 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-trusted-ca-bundle\") pod \"periodic-gathering-zhmhr-2v9dv\" (UID: \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\") " pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" Oct 30 18:33:52.290600 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.290143 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-archives-path\") pod \"periodic-gathering-zhmhr-2v9dv\" (UID: \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\") " pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" Oct 30 18:33:52.290600 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.290167 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rswb\" (UniqueName: \"kubernetes.io/projected/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-kube-api-access-5rswb\") pod \"periodic-gathering-zhmhr-2v9dv\" (UID: \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\") " pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" Oct 30 18:33:52.290754 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.290729 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-archives-path\") pod \"periodic-gathering-zhmhr-2v9dv\" (UID: \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\") " pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" Oct 30 18:33:52.290798 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.290741 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-service-ca-bundle\") pod \"periodic-gathering-zhmhr-2v9dv\" (UID: \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\") " pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" Oct 30 18:33:52.291368 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.291346 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-trusted-ca-bundle\") pod \"periodic-gathering-zhmhr-2v9dv\" (UID: \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\") " pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" Oct 30 18:33:52.352448 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.352414 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rswb\" (UniqueName: \"kubernetes.io/projected/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-kube-api-access-5rswb\") pod \"periodic-gathering-zhmhr-2v9dv\" (UID: \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\") " pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" Oct 30 18:33:52.418897 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.418850 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" Oct 30 18:33:52.592944 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.592837 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:33:52.592944 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.592886 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af0639e4-bb41-4099-a9cf-03678c968c0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:33:52.593214 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:52.592978 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 18:33:52.593214 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:52.593023 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 18:33:52.593214 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:52.593034 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af0639e4-bb41-4099-a9cf-03678c968c0b-kube-rbac-proxy-cm podName:af0639e4-bb41-4099-a9cf-03678c968c0b nodeName:}" failed. No retries permitted until 2026-10-30 18:33:53.593020377 +0000 UTC m=+34.040729040 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/af0639e4-bb41-4099-a9cf-03678c968c0b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v9jf7" (UID: "af0639e4-bb41-4099-a9cf-03678c968c0b") : configmap "kube-rbac-proxy" not found Oct 30 18:33:52.593214 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:52.593105 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls podName:af0639e4-bb41-4099-a9cf-03678c968c0b nodeName:}" failed. No retries permitted until 2026-10-30 18:33:53.593087286 +0000 UTC m=+34.040795952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9jf7" (UID: "af0639e4-bb41-4099-a9cf-03678c968c0b") : secret "insights-runtime-extractor-tls" not found Oct 30 18:33:52.791692 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.791656 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-zhmhr-2v9dv"] Oct 30 18:33:52.795010 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:52.794967 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f6da88_0f08_47df_aa8f_7594b0dd3cc3.slice/crio-44d18475766b206efd944f4e66107fafe7cc348b214cd44c7a889f8a58b44789 WatchSource:0}: Error finding container 44d18475766b206efd944f4e66107fafe7cc348b214cd44c7a889f8a58b44789: Status 404 returned error can't find the container with id 44d18475766b206efd944f4e66107fafe7cc348b214cd44c7a889f8a58b44789 Oct 30 18:33:52.995814 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.995776 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84bbebc-7b6d-4e22-aacd-c98586717109-metrics-certs\") pod \"network-metrics-daemon-flgwv\" (UID: \"c84bbebc-7b6d-4e22-aacd-c98586717109\") " pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:33:52.995986 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.995891 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/429722d3-91b1-4c92-b78e-19a12bc5bac9-original-pull-secret\") pod \"global-pull-secret-syncer-frkss\" (UID: \"429722d3-91b1-4c92-b78e-19a12bc5bac9\") " pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:52.996066 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:52.995975 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 18:33:52.996122 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:52.996066 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c84bbebc-7b6d-4e22-aacd-c98586717109-metrics-certs podName:c84bbebc-7b6d-4e22-aacd-c98586717109 nodeName:}" failed. No retries permitted until 2026-10-30 18:34:24.99604583 +0000 UTC m=+65.443754498 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c84bbebc-7b6d-4e22-aacd-c98586717109-metrics-certs") pod "network-metrics-daemon-flgwv" (UID: "c84bbebc-7b6d-4e22-aacd-c98586717109") : secret "metrics-daemon-secret" not found Oct 30 18:33:52.998903 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:52.998870 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/429722d3-91b1-4c92-b78e-19a12bc5bac9-original-pull-secret\") pod \"global-pull-secret-syncer-frkss\" (UID: \"429722d3-91b1-4c92-b78e-19a12bc5bac9\") " pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:53.097025 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:53.096990 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxxl5\" (UniqueName: \"kubernetes.io/projected/17dae92a-ae2b-4c09-8001-48a81a8cf809-kube-api-access-mxxl5\") pod \"network-check-target-hh4r5\" (UID: \"17dae92a-ae2b-4c09-8001-48a81a8cf809\") " pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:53.100072 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:53.100045 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxxl5\" (UniqueName: \"kubernetes.io/projected/17dae92a-ae2b-4c09-8001-48a81a8cf809-kube-api-access-mxxl5\") pod \"network-check-target-hh4r5\" (UID: \"17dae92a-ae2b-4c09-8001-48a81a8cf809\") " pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:53.124064 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:53.124028 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:33:53.283869 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:53.283756 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" event={"ID":"10f6da88-0f08-47df-aa8f-7594b0dd3cc3","Type":"ContainerStarted","Data":"44d18475766b206efd944f4e66107fafe7cc348b214cd44c7a889f8a58b44789"} Oct 30 18:33:53.397473 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:53.397441 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:53.502872 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:53.502784 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-frkss"] Oct 30 18:33:53.507121 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:53.507086 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429722d3_91b1_4c92_b78e_19a12bc5bac9.slice/crio-874e0ce10c321522a4ed6c5e132613dac0d85a8d2e24163cd160711c594f209b WatchSource:0}: Error finding container 874e0ce10c321522a4ed6c5e132613dac0d85a8d2e24163cd160711c594f209b: Status 404 returned error can't find the container with id 874e0ce10c321522a4ed6c5e132613dac0d85a8d2e24163cd160711c594f209b Oct 30 18:33:53.600567 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:53.600457 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:33:53.600567 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:53.600539 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af0639e4-bb41-4099-a9cf-03678c968c0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:33:53.600786 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:53.600628 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 18:33:53.600786 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:53.600651 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 18:33:53.600786 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:53.600708 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls podName:af0639e4-bb41-4099-a9cf-03678c968c0b nodeName:}" failed. No retries permitted until 2026-10-30 18:33:55.600689754 +0000 UTC m=+36.048398417 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9jf7" (UID: "af0639e4-bb41-4099-a9cf-03678c968c0b") : secret "insights-runtime-extractor-tls" not found Oct 30 18:33:53.600786 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:53.600731 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af0639e4-bb41-4099-a9cf-03678c968c0b-kube-rbac-proxy-cm podName:af0639e4-bb41-4099-a9cf-03678c968c0b nodeName:}" failed. No retries permitted until 2026-10-30 18:33:55.600718642 +0000 UTC m=+36.048427342 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/af0639e4-bb41-4099-a9cf-03678c968c0b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v9jf7" (UID: "af0639e4-bb41-4099-a9cf-03678c968c0b") : configmap "kube-rbac-proxy" not found Oct 30 18:33:53.643573 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:53.643542 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vd4zw"] Oct 30 18:33:53.646150 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:53.646107 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vd4zw" Oct 30 18:33:53.701771 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:53.701736 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmz4t\" (UniqueName: \"kubernetes.io/projected/4ac7c1cf-9559-4281-be30-485e5e181d61-kube-api-access-cmz4t\") pod \"migrator-66df5d496b-vd4zw\" (UID: \"4ac7c1cf-9559-4281-be30-485e5e181d61\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vd4zw" Oct 30 18:33:53.701952 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:53.701793 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:33:53.701952 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:53.701944 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 18:33:53.702056 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:53.702005 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls podName:eaa9cd18-eea8-4b24-b6af-c8cd6e278011 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:57.701986551 +0000 UTC m=+38.149695233 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls") pod "dns-default-49jb4" (UID: "eaa9cd18-eea8-4b24-b6af-c8cd6e278011") : secret "dns-default-metrics-tls" not found Oct 30 18:33:53.704789 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:53.704761 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 18:33:53.704953 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:53.704869 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 18:33:53.714260 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:53.710728 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-phb6k\"" Oct 30 18:33:53.714260 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:53.714174 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vd4zw"] Oct 30 18:33:53.784720 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:53.784688 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hh4r5"] Oct 30 18:33:53.789452 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:53.789421 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17dae92a_ae2b_4c09_8001_48a81a8cf809.slice/crio-aef8fe1f0bf8712e8a0975dfd9d83b032f9483336053dcd36ac79d96492ee3b3 WatchSource:0}: Error finding container aef8fe1f0bf8712e8a0975dfd9d83b032f9483336053dcd36ac79d96492ee3b3: Status 404 returned error can't find the container with id aef8fe1f0bf8712e8a0975dfd9d83b032f9483336053dcd36ac79d96492ee3b3 Oct 30 18:33:53.802811 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:53.802769 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmz4t\" (UniqueName: \"kubernetes.io/projected/4ac7c1cf-9559-4281-be30-485e5e181d61-kube-api-access-cmz4t\") pod \"migrator-66df5d496b-vd4zw\" (UID: \"4ac7c1cf-9559-4281-be30-485e5e181d61\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vd4zw" Oct 30 18:33:53.868380 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:53.868314 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmz4t\" (UniqueName: \"kubernetes.io/projected/4ac7c1cf-9559-4281-be30-485e5e181d61-kube-api-access-cmz4t\") pod \"migrator-66df5d496b-vd4zw\" (UID: \"4ac7c1cf-9559-4281-be30-485e5e181d61\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vd4zw" Oct 30 18:33:53.959846 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:53.959800 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vd4zw" Oct 30 18:33:54.287973 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:54.287938 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-frkss" event={"ID":"429722d3-91b1-4c92-b78e-19a12bc5bac9","Type":"ContainerStarted","Data":"874e0ce10c321522a4ed6c5e132613dac0d85a8d2e24163cd160711c594f209b"} Oct 30 18:33:54.290505 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:54.289989 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hjrzq" event={"ID":"886a6628-cf58-4784-85e0-d1ca14a2403c","Type":"ContainerStarted","Data":"1a6db802432c81d81d89471dd77122c4c069adfdeb05fc4e45c15fa432212198"} Oct 30 18:33:54.291274 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:54.291247 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hh4r5" event={"ID":"17dae92a-ae2b-4c09-8001-48a81a8cf809","Type":"ContainerStarted","Data":"aef8fe1f0bf8712e8a0975dfd9d83b032f9483336053dcd36ac79d96492ee3b3"} Oct 30 18:33:54.345865 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:54.345834 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vd4zw"] Oct 30 18:33:54.349548 ip-10-0-133-56 kubenswrapper[2571]: W1030 18:33:54.349510 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac7c1cf_9559_4281_be30_485e5e181d61.slice/crio-d719deca4e514317b4ed3d57beec0318ecf1d28ab3c74f9e38c3ad58e869af1b WatchSource:0}: Error finding container d719deca4e514317b4ed3d57beec0318ecf1d28ab3c74f9e38c3ad58e869af1b: Status 404 returned error can't find the container with id d719deca4e514317b4ed3d57beec0318ecf1d28ab3c74f9e38c3ad58e869af1b Oct 30 18:33:54.410565 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:54.410499 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hjrzq" podStartSLOduration=2.104970617 podStartE2EDuration="5.410483709s" totalImagesPullingTime="3.305513092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:54.409767104 +0000 UTC m=+34.857475794" watchObservedRunningTime="2026-10-30 18:33:54.410483709 +0000 UTC m=+34.858192393" Oct 30 18:33:54.712683 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:54.712530 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-7s7lj"] Oct 30 18:33:54.714925 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:54.714887 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-7s7lj" Oct 30 18:33:54.768211 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:54.768120 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 18:33:54.768211 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:54.768175 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 18:33:54.768211 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:54.768176 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 18:33:54.768518 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:54.768222 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 18:33:54.768518 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:54.768279 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9j6lr\"" Oct 30 18:33:54.774878 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:54.774837 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-7s7lj"] Oct 30 18:33:54.812481 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:54.812403 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2173c7a3-6b8f-4df7-8c17-cfd81932930a-signing-key\") pod \"service-ca-5fffc97bdf-7s7lj\" (UID: \"2173c7a3-6b8f-4df7-8c17-cfd81932930a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7s7lj" Oct 30 18:33:54.812729 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:54.812572 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2173c7a3-6b8f-4df7-8c17-cfd81932930a-signing-cabundle\") pod \"service-ca-5fffc97bdf-7s7lj\" (UID: \"2173c7a3-6b8f-4df7-8c17-cfd81932930a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7s7lj" Oct 30 18:33:54.812729 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:54.812638 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lllm7\" (UniqueName: \"kubernetes.io/projected/2173c7a3-6b8f-4df7-8c17-cfd81932930a-kube-api-access-lllm7\") pod \"service-ca-5fffc97bdf-7s7lj\" (UID: \"2173c7a3-6b8f-4df7-8c17-cfd81932930a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7s7lj" Oct 30 18:33:54.914773 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:54.913898 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2173c7a3-6b8f-4df7-8c17-cfd81932930a-signing-cabundle\") pod \"service-ca-5fffc97bdf-7s7lj\" (UID: \"2173c7a3-6b8f-4df7-8c17-cfd81932930a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7s7lj" Oct 30 18:33:54.914773 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:54.913985 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lllm7\" (UniqueName: \"kubernetes.io/projected/2173c7a3-6b8f-4df7-8c17-cfd81932930a-kube-api-access-lllm7\") pod \"service-ca-5fffc97bdf-7s7lj\" (UID: \"2173c7a3-6b8f-4df7-8c17-cfd81932930a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7s7lj" Oct 30 18:33:54.914773 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:54.914032 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2173c7a3-6b8f-4df7-8c17-cfd81932930a-signing-key\") pod \"service-ca-5fffc97bdf-7s7lj\" (UID: \"2173c7a3-6b8f-4df7-8c17-cfd81932930a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7s7lj" Oct 30 18:33:54.914773 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:54.914710 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2173c7a3-6b8f-4df7-8c17-cfd81932930a-signing-cabundle\") pod \"service-ca-5fffc97bdf-7s7lj\" (UID: \"2173c7a3-6b8f-4df7-8c17-cfd81932930a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7s7lj" Oct 30 18:33:54.917506 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:54.917477 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2173c7a3-6b8f-4df7-8c17-cfd81932930a-signing-key\") pod \"service-ca-5fffc97bdf-7s7lj\" (UID: \"2173c7a3-6b8f-4df7-8c17-cfd81932930a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7s7lj" Oct 30 18:33:54.978067 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:54.977987 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lllm7\" (UniqueName: \"kubernetes.io/projected/2173c7a3-6b8f-4df7-8c17-cfd81932930a-kube-api-access-lllm7\") pod \"service-ca-5fffc97bdf-7s7lj\" (UID: \"2173c7a3-6b8f-4df7-8c17-cfd81932930a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7s7lj" Oct 30 18:33:55.028225 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:55.028190 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-7s7lj" Oct 30 18:33:55.296748 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:55.296674 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vd4zw" event={"ID":"4ac7c1cf-9559-4281-be30-485e5e181d61","Type":"ContainerStarted","Data":"d719deca4e514317b4ed3d57beec0318ecf1d28ab3c74f9e38c3ad58e869af1b"} Oct 30 18:33:55.416776 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:55.416683 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-7s7lj"] Oct 30 18:33:55.621994 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:55.621910 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:33:55.621994 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:55.621967 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af0639e4-bb41-4099-a9cf-03678c968c0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:33:55.622276 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:55.622057 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 18:33:55.622276 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:55.622109 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af0639e4-bb41-4099-a9cf-03678c968c0b-kube-rbac-proxy-cm podName:af0639e4-bb41-4099-a9cf-03678c968c0b nodeName:}" failed. No retries permitted until 2026-10-30 18:33:59.622093012 +0000 UTC m=+40.069801675 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/af0639e4-bb41-4099-a9cf-03678c968c0b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v9jf7" (UID: "af0639e4-bb41-4099-a9cf-03678c968c0b") : configmap "kube-rbac-proxy" not found Oct 30 18:33:55.622276 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:55.622242 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 18:33:55.622276 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:55.622273 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls podName:af0639e4-bb41-4099-a9cf-03678c968c0b nodeName:}" failed. No retries permitted until 2026-10-30 18:33:59.622265283 +0000 UTC m=+40.069973946 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9jf7" (UID: "af0639e4-bb41-4099-a9cf-03678c968c0b") : secret "insights-runtime-extractor-tls" not found Oct 30 18:33:56.302108 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:56.302009 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-7s7lj" event={"ID":"2173c7a3-6b8f-4df7-8c17-cfd81932930a","Type":"ContainerStarted","Data":"b76f3136135467860e65090a3b9d6d4f4bd4f5f84e6f6e4466c5e10e42af1155"} Oct 30 18:33:56.304950 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:56.304438 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" event={"ID":"10f6da88-0f08-47df-aa8f-7594b0dd3cc3","Type":"ContainerStarted","Data":"dd0e0940c9df298df8aa64d46bcfaf3df7c7a33ebd904bd673f277a3179d4cc7"} Oct 30 18:33:57.309169 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:57.309113 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vd4zw" event={"ID":"4ac7c1cf-9559-4281-be30-485e5e181d61","Type":"ContainerStarted","Data":"156e64dc83205507911910f919e2613e56cfbf3706b583800383a70d62c153c1"} Oct 30 18:33:57.433560 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:57.433499 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" podStartSLOduration=2.261927064 podStartE2EDuration="5.433483775s" totalImagesPullingTime="3.171556711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:56.428195862 +0000 UTC m=+36.875904548" watchObservedRunningTime="2026-10-30 18:33:57.433483775 +0000 UTC m=+37.881192472" Oct 30 18:33:57.433780 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:57.433609 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vd4zw" podStartSLOduration=2.316163875 podStartE2EDuration="4.433600264s" totalImagesPullingTime="2.117436389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:57.432268644 +0000 UTC m=+37.879977331" watchObservedRunningTime="2026-10-30 18:33:57.433600264 +0000 UTC m=+37.881308949" Oct 30 18:33:57.742667 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:57.742035 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:33:57.742667 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:57.742230 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 18:33:57.742667 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:57.742304 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls podName:eaa9cd18-eea8-4b24-b6af-c8cd6e278011 nodeName:}" failed. No retries permitted until 2026-10-30 18:34:05.742285863 +0000 UTC m=+46.189994529 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls") pod "dns-default-49jb4" (UID: "eaa9cd18-eea8-4b24-b6af-c8cd6e278011") : secret "dns-default-metrics-tls" not found Oct 30 18:33:58.314289 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:58.314250 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hh4r5" event={"ID":"17dae92a-ae2b-4c09-8001-48a81a8cf809","Type":"ContainerStarted","Data":"14a342981684ceff7326bd9431b3e6a4671dfad5c4e730859d353b0839311c25"} Oct 30 18:33:58.430241 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:58.430170 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hh4r5" podStartSLOduration=34.305373923 podStartE2EDuration="38.430147875s" totalImagesPullingTime="4.124773952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:58.429944722 +0000 UTC m=+38.877653408" watchObservedRunningTime="2026-10-30 18:33:58.430147875 +0000 UTC m=+38.877856559" Oct 30 18:33:59.318971 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:59.318923 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-7s7lj" event={"ID":"2173c7a3-6b8f-4df7-8c17-cfd81932930a","Type":"ContainerStarted","Data":"b84fd09379f358da75cfc783bd3290467f1207fdf3adac2e06db6c41603c2164"} Oct 30 18:33:59.319509 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:59.319152 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:33:59.436894 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:59.436836 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-7s7lj" podStartSLOduration=2.00790858 podStartE2EDuration="5.436820085s" totalImagesPullingTime="3.428911505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:59.436467034 +0000 UTC m=+39.884175719" watchObservedRunningTime="2026-10-30 18:33:59.436820085 +0000 UTC m=+39.884528770" Oct 30 18:33:59.549287 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:59.549260 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7jq9m_2d258432-7e24-4b20-a0e8-258aa1e1a4a5/dns-node-resolver/0.log" Oct 30 18:33:59.660564 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:59.660524 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af0639e4-bb41-4099-a9cf-03678c968c0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:33:59.660741 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:59.660636 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:33:59.660741 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:59.660730 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 18:33:59.660827 ip-10-0-133-56 kubenswrapper[2571]: E1030 18:33:59.660780 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls podName:af0639e4-bb41-4099-a9cf-03678c968c0b nodeName:}" failed. No retries permitted until 2026-10-30 18:34:07.660767349 +0000 UTC m=+48.108476025 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9jf7" (UID: "af0639e4-bb41-4099-a9cf-03678c968c0b") : secret "insights-runtime-extractor-tls" not found Oct 30 18:33:59.661236 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:59.661209 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af0639e4-bb41-4099-a9cf-03678c968c0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:33:59.895878 ip-10-0-133-56 kubenswrapper[2571]: I1030 18:33:59.895669 2571 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:33:59.895814 ip-10-0-133-56 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 18:33:59.906862 ip-10-0-133-56 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 18:33:59.907097 ip-10-0-133-56 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 18:33:59.907213 ip-10-0-133-56 systemd[1]: kubelet.service: Consumed 2.696s CPU time. Oct 30 18:33:59.925993 ip-10-0-133-56 systemd[1]: Starting Kubernetes Kubelet... Oct 30 18:34:00.008928 ip-10-0-133-56 kubenswrapper[4531]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:34:00.008928 ip-10-0-133-56 kubenswrapper[4531]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 18:34:00.008928 ip-10-0-133-56 kubenswrapper[4531]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:34:00.008928 ip-10-0-133-56 kubenswrapper[4531]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:34:00.011333 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011231 4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:34:00.011333 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011332 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:34:00.011333 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011336 4531 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:34:00.011333 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011339 4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:34:00.011472 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011342 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:34:00.011472 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011346 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:34:00.011472 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011351 4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:34:00.011472 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011355 4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:34:00.011472 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011360 4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:34:00.011472 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011362 4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:34:00.011472 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011365 4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:34:00.011472 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011368 4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:34:00.011472 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011370 4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:34:00.011472 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011373 4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:34:00.011472 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011376 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:34:00.011472 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011379 4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:34:00.011472 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011382 4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:34:00.011472 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011384 4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:34:00.011472 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011387 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:34:00.011472 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011416 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:34:00.011472 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011420 4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:34:00.011472 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011424 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:34:00.011472 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011427 4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:34:00.011940 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011431 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:34:00.011940 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011433 4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:34:00.011940 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011436 4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:34:00.011940 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011438 4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:34:00.011940 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011441 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:34:00.011940 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011444 4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:34:00.011940 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011452 4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:34:00.011940 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011455 4531 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:34:00.011940 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011458 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:34:00.011940 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011461 4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:34:00.011940 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011463 4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:34:00.011940 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011467 4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:34:00.011940 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011469 4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:34:00.011940 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011472 4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:34:00.011940 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011475 4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:34:00.011940 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011477 4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:34:00.011940 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011480 4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:34:00.011940 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011484 4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:34:00.011940 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011487 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:34:00.011940 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011490 4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:34:00.012468 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011492 4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:34:00.012468 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011495 4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:34:00.012468 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011497 4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:34:00.012468 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011500 4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:34:00.012468 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011504 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:34:00.012468 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011507 4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:34:00.012468 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011510 4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:34:00.012468 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011512 4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:34:00.012468 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011515 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:34:00.012468 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011517 4531 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:34:00.012468 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011520 4531 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:34:00.012468 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011522 4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:34:00.012468 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011525 4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:34:00.012468 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011528 4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:34:00.012468 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011530 4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:34:00.012468 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011533 4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:34:00.012468 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011536 4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:34:00.012468 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011539 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:34:00.012468 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011542 4531 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:34:00.012468 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011544 4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:34:00.012963 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011547 4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:34:00.012963 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011550 4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:34:00.012963 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011552 4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:34:00.012963 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011556 4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:34:00.012963 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011560 4531 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:34:00.012963 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011563 4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:34:00.012963 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011566 4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:34:00.012963 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011568 4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:34:00.012963 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011571 4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:34:00.012963 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011574 4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:34:00.012963 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011577 4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:34:00.012963 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011579 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:34:00.012963 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011582 4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:34:00.012963 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011587 4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:34:00.012963 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011590 4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:34:00.012963 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011593 4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:34:00.012963 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011595 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:34:00.012963 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011598 4531 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:34:00.012963 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011601 4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:34:00.012963 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011604 4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:34:00.013487 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011606 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:34:00.013487 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011609 4531 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:34:00.013487 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011611 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:34:00.013487 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011614 4531 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:34:00.013487 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011616 4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:34:00.013487 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011620 4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:34:00.013487 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011622 4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:34:00.013487 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011625 4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:34:00.013487 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011627 4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:34:00.013487 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011630 4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:34:00.013487 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011633 4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:34:00.013487 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011636 4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:34:00.013487 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011638 4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:34:00.013487 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011641 4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:34:00.013487 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011643 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:34:00.013487 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011646 4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:34:00.013487 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011649 4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:34:00.013487 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011651 4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:34:00.013487 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011654 4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:34:00.013487 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011656 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:34:00.013996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011659 4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:34:00.013996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011661 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:34:00.013996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.011665 4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:34:00.013996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012530 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:34:00.013996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012539 4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:34:00.013996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012543 4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:34:00.013996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012546 4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:34:00.013996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012548 4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:34:00.013996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012551 4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:34:00.013996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012554 4531 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:34:00.013996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012557 4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:34:00.013996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012560 4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:34:00.013996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012563 4531 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:34:00.013996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012566 4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:34:00.013996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012569 4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:34:00.013996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012571 4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:34:00.013996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012574 4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:34:00.013996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012577 4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:34:00.013996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012582 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:34:00.013996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012585 4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:34:00.014564 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012588 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:34:00.014564 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012590 4531 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:34:00.014564 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012593 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:34:00.014564 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012595 4531 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:34:00.014564 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012598 4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:34:00.014564 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012601 4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:34:00.014564 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012604 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:34:00.014564 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012607 4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:34:00.014564 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012609 4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:34:00.014564 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012612 4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:34:00.014564 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012615 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:34:00.014564 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012617 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:34:00.014564 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012621 4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:34:00.014564 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012626 4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:34:00.014564 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012629 4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:34:00.014564 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012632 4531 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:34:00.014564 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012634 4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:34:00.014564 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012637 4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:34:00.014564 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012639 4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:34:00.015037 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012642 4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:34:00.015037 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012645 4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:34:00.015037 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012647 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:34:00.015037 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012649 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:34:00.015037 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012653 4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:34:00.015037 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012655 4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:34:00.015037 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012658 4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:34:00.015037 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012660 4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:34:00.015037 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012663 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:34:00.015037 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012665 4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:34:00.015037 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012667 4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:34:00.015037 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012673 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:34:00.015037 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012676 4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:34:00.015037 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012678 4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:34:00.015037 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012681 4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:34:00.015037 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012683 4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:34:00.015037 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012686 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:34:00.015037 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012689 4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:34:00.015037 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012691 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:34:00.015532 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012693 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:34:00.015532 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012696 4531 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:34:00.015532 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012699 4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:34:00.015532 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012701 4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:34:00.015532 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012703 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:34:00.015532 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012706 4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:34:00.015532 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012708 4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:34:00.015532 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012713 4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:34:00.015532 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012715 4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:34:00.015532 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012718 4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:34:00.015532 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012720 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:34:00.015532 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012723 4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:34:00.015532 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012725 4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:34:00.015532 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012728 4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:34:00.015532 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012730 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:34:00.015532 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012733 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:34:00.015532 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012736 4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:34:00.015532 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012739 4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:34:00.015532 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012742 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:34:00.016013 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012745 4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:34:00.016013 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012747 4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:34:00.016013 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012749 4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:34:00.016013 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012752 4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:34:00.016013 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012754 4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:34:00.016013 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012758 4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:34:00.016013 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012761 4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:34:00.016013 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012764 4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:34:00.016013 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012766 4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:34:00.016013 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012769 4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:34:00.016013 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012772 4531 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:34:00.016013 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012775 4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:34:00.016013 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012777 4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:34:00.016013 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012780 4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:34:00.016013 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012782 4531 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:34:00.016013 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012785 4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:34:00.016013 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012787 4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:34:00.016013 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012790 4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:34:00.016013 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012792 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:34:00.016503 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012795 4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:34:00.016503 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012797 4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:34:00.016503 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012800 4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:34:00.016503 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012802 4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:34:00.016503 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012805 4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:34:00.016503 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012807 4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:34:00.016503 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012811 4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:34:00.016503 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012815 4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:34:00.016503 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012818 4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:34:00.016503 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012821 4531 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:34:00.016503 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012824 4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:34:00.016503 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012826 4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:34:00.016503 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.012829 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:34:00.016503 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013398 4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:34:00.016503 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013404 4531 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:34:00.016503 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013407 4531 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:34:00.016503 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013410 4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:34:00.016503 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013413 4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:34:00.016503 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013419 4531 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:34:00.016503 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013423 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:34:00.016998 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013426 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:34:00.016998 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013428 4531 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:34:00.016998 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013433 4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:34:00.016998 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013436 4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:34:00.016998 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013440 4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:34:00.016998 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013443 4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:34:00.016998 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013446 4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:34:00.016998 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013448 4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:34:00.016998 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013451 4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:34:00.016998 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013453 4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:34:00.016998 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013456 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:34:00.016998 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013459 4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:34:00.016998 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013461 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:34:00.016998 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013466 4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:34:00.016998 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013470 4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:34:00.016998 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013473 4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:34:00.016998 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013475 4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:34:00.016998 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013478 4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:34:00.016998 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013483 4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:34:00.017495 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013486 4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:34:00.017495 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013488 4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:34:00.017495 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013491 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:34:00.017495 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013494 4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:34:00.017495 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013496 4531 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:34:00.017495 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013499 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:34:00.017495 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013503 4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:34:00.017495 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013506 4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:34:00.017495 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013508 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:34:00.017495 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013510 4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:34:00.017495 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013513 4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:34:00.017495 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013516 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:34:00.017495 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013519 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:34:00.017495 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013521 4531 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:34:00.017495 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013524 4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:34:00.017495 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013526 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:34:00.017495 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013529 4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:34:00.017495 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013531 4531 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:34:00.017495 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013534 4531 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:34:00.017495 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013536 4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:34:00.018054 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013539 4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:34:00.018054 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013541 4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:34:00.018054 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013544 4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:34:00.018054 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013546 4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:34:00.018054 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013549 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:34:00.018054 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013551 4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:34:00.018054 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013554 4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:34:00.018054 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013557 4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:34:00.018054 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013559 4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:34:00.018054 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013562 4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:34:00.018054 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013564 4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:34:00.018054 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013568 4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:34:00.018054 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013571 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:34:00.018054 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013573 4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:34:00.018054 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013576 4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:34:00.018054 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013578 4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:34:00.018054 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013581 4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:34:00.018054 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013584 4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:34:00.018054 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013586 4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:34:00.018054 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013590 4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:34:00.018573 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013592 4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:34:00.018573 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013595 4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:34:00.018573 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013597 4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:34:00.018573 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013600 4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:34:00.018573 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013603 4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:34:00.018573 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013606 4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:34:00.018573 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013608 4531 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:34:00.018573 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013611 4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:34:00.018573 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013614 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:34:00.018573 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013616 4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:34:00.018573 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013619 4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:34:00.018573 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013621 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:34:00.018573 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013624 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:34:00.018573 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013626 4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:34:00.018573 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013629 4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:34:00.018573 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013631 4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:34:00.018573 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013633 4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:34:00.018573 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013636 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:34:00.018573 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013639 4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:34:00.019051 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013641 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:34:00.019051 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013644 4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:34:00.019051 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013646 4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:34:00.019051 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013649 4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:34:00.019051 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013652 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:34:00.019051 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013655 4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:34:00.019051 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013657 4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:34:00.019051 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013660 4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:34:00.019051 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013662 4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:34:00.019051 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013665 4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:34:00.019051 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013667 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:34:00.019051 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013670 4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:34:00.019051 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013683 4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:34:00.019051 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013687 4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:34:00.019051 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013690 4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:34:00.019051 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013693 4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:34:00.019051 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013696 4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:34:00.019051 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013700 4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:34:00.019051 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013702 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:34:00.019051 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013705 4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:34:00.019732 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.013707 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:34:00.019732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013821 4531 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 18:34:00.019732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013833 4531 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 18:34:00.019732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013842 4531 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 18:34:00.019732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013848 4531 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 18:34:00.019732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013852 4531 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 18:34:00.019732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013856 4531 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 18:34:00.019732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013860 4531 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 18:34:00.019732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013864 4531 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 18:34:00.019732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013868 4531 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 18:34:00.019732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013871 4531 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 18:34:00.019732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013874 4531 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 18:34:00.019732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013878 4531 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 18:34:00.019732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013881 4531 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 18:34:00.019732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013884 4531 flags.go:64] FLAG: --cgroup-root="" Oct 30 18:34:00.019732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013887 4531 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 18:34:00.019732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013893 4531 flags.go:64] FLAG: --client-ca-file="" Oct 30 18:34:00.019732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013896 4531 flags.go:64] FLAG: --cloud-provider="external" Oct 30 18:34:00.019732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013899 4531 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 18:34:00.019732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013904 4531 flags.go:64] FLAG: --cluster-domain="" Oct 30 18:34:00.019732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013908 4531 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 18:34:00.019732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013911 4531 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 18:34:00.019732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013914 4531 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 18:34:00.019732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013918 4531 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 18:34:00.020350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013922 4531 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 18:34:00.020350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013925 4531 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 18:34:00.020350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013929 4531 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 18:34:00.020350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013933 4531 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 18:34:00.020350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013936 4531 flags.go:64] FLAG: --contention-profiling="false" Oct 30 18:34:00.020350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013939 4531 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 18:34:00.020350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013942 4531 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 18:34:00.020350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013945 4531 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 18:34:00.020350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013948 4531 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 18:34:00.020350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013952 4531 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 18:34:00.020350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013955 4531 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 18:34:00.020350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013959 4531 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 18:34:00.020350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013962 4531 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 18:34:00.020350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013965 4531 flags.go:64] FLAG: --enable-server="true" Oct 30 18:34:00.020350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013967 4531 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 18:34:00.020350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013972 4531 flags.go:64] FLAG: --event-burst="100" Oct 30 18:34:00.020350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013975 4531 flags.go:64] FLAG: --event-qps="50" Oct 30 18:34:00.020350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013978 4531 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 18:34:00.020350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013981 4531 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 18:34:00.020350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013984 4531 flags.go:64] FLAG: --eviction-hard="" Oct 30 18:34:00.020350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013989 4531 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 18:34:00.020350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013992 4531 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 18:34:00.020350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013995 4531 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 18:34:00.020350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.013998 4531 flags.go:64] FLAG: --eviction-soft="" Oct 30 18:34:00.020350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014003 4531 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 18:34:00.021046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014006 4531 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 18:34:00.021046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014009 4531 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 18:34:00.021046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014012 4531 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 18:34:00.021046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014017 4531 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 18:34:00.021046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014020 4531 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 18:34:00.021046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014022 4531 flags.go:64] FLAG: --feature-gates="" Oct 30 18:34:00.021046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014030 4531 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 18:34:00.021046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014033 4531 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 18:34:00.021046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014037 4531 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 18:34:00.021046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014040 4531 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 18:34:00.021046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014044 4531 flags.go:64] FLAG: --healthz-port="10248" Oct 30 18:34:00.021046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014047 4531 flags.go:64] FLAG: --help="false" Oct 30 18:34:00.021046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014051 4531 flags.go:64] FLAG: --hostname-override="ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:34:00.021046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014054 4531 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 18:34:00.021046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014057 4531 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 18:34:00.021046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014060 4531 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 18:34:00.021046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014064 4531 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 18:34:00.021046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014067 4531 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 18:34:00.021046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014070 4531 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 18:34:00.021046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014073 4531 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 18:34:00.021046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014076 4531 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 18:34:00.021046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014079 4531 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 18:34:00.021046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014082 4531 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 18:34:00.021652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014085 4531 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 18:34:00.021652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014088 4531 flags.go:64] FLAG: --kube-reserved="" Oct 30 18:34:00.021652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014091 4531 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 18:34:00.021652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014094 4531 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 18:34:00.021652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014097 4531 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 18:34:00.021652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014100 4531 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 18:34:00.021652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014103 4531 flags.go:64] FLAG: --lock-file="" Oct 30 18:34:00.021652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014106 4531 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 18:34:00.021652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014111 4531 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 18:34:00.021652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014114 4531 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 18:34:00.021652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014119 4531 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 18:34:00.021652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014122 4531 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 18:34:00.021652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014145 4531 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 18:34:00.021652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014150 4531 flags.go:64] FLAG: --logging-format="text" Oct 30 18:34:00.021652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014154 4531 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 18:34:00.021652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014160 4531 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 18:34:00.021652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014164 4531 flags.go:64] FLAG: --manifest-url="" Oct 30 18:34:00.021652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014167 4531 flags.go:64] FLAG: --manifest-url-header="" Oct 30 18:34:00.021652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014173 4531 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 18:34:00.021652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014176 4531 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 18:34:00.021652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014181 4531 flags.go:64] FLAG: --max-pods="110" Oct 30 18:34:00.021652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014185 4531 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 18:34:00.021652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014187 4531 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 18:34:00.021652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014191 4531 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 18:34:00.021652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014194 4531 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 18:34:00.022602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014197 4531 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 18:34:00.022602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014200 4531 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 18:34:00.022602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014203 4531 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 18:34:00.022602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014211 4531 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 18:34:00.022602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014214 4531 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 18:34:00.022602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014217 4531 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 18:34:00.022602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014220 4531 flags.go:64] FLAG: --pod-cidr="" Oct 30 18:34:00.022602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014223 4531 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 18:34:00.022602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014226 4531 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 18:34:00.022602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014229 4531 flags.go:64] FLAG: --pods-per-core="0" Oct 30 18:34:00.022602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014232 4531 flags.go:64] FLAG: --port="10250" Oct 30 18:34:00.022602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014235 4531 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 18:34:00.022602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014238 4531 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0badc0c146ab7f8f0" Oct 30 18:34:00.022602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014241 4531 flags.go:64] FLAG: --qos-reserved="" Oct 30 18:34:00.022602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014245 4531 flags.go:64] FLAG: --read-only-port="10255" Oct 30 18:34:00.022602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014249 4531 flags.go:64] FLAG: --register-node="true" Oct 30 18:34:00.022602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014252 4531 flags.go:64] FLAG: --register-with-taints="" Oct 30 18:34:00.022602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014256 4531 flags.go:64] FLAG: --registry-burst="10" Oct 30 18:34:00.022602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014259 4531 flags.go:64] FLAG: --registry-qps="5" Oct 30 18:34:00.022602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014264 4531 flags.go:64] FLAG: --reserved-cpus="" Oct 30 18:34:00.022602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014267 4531 flags.go:64] FLAG: --reserved-memory="" Oct 30 18:34:00.022602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014271 4531 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 18:34:00.022602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014275 4531 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 18:34:00.022602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014278 4531 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 18:34:00.022602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014281 4531 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 18:34:00.022602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014284 4531 flags.go:64] FLAG: --runonce="false" Oct 30 18:34:00.023589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014286 4531 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 18:34:00.023589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014290 4531 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 18:34:00.023589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014293 4531 flags.go:64] FLAG: --seccomp-default="false" Oct 30 18:34:00.023589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014297 4531 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 18:34:00.023589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014300 4531 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 18:34:00.023589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014303 4531 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 18:34:00.023589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014307 4531 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 18:34:00.023589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014310 4531 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 18:34:00.023589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014313 4531 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 18:34:00.023589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014317 4531 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 18:34:00.023589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014320 4531 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 18:34:00.023589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014323 4531 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 18:34:00.023589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014326 4531 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 18:34:00.023589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014329 4531 flags.go:64] FLAG: --system-cgroups="" Oct 30 18:34:00.023589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014332 4531 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 18:34:00.023589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014337 4531 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 18:34:00.023589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014340 4531 flags.go:64] FLAG: --tls-cert-file="" Oct 30 18:34:00.023589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014343 4531 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 18:34:00.023589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014351 4531 flags.go:64] FLAG: --tls-min-version="" Oct 30 18:34:00.023589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014354 4531 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 18:34:00.023589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014357 4531 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 18:34:00.023589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014362 4531 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 18:34:00.023589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014365 4531 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 18:34:00.023589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014368 4531 flags.go:64] FLAG: --v="2" Oct 30 18:34:00.023589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014373 4531 flags.go:64] FLAG: --version="false" Oct 30 18:34:00.024631 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014382 4531 flags.go:64] FLAG: --vmodule="" Oct 30 18:34:00.024631 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014386 4531 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 18:34:00.024631 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014390 4531 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 18:34:00.024631 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014545 4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:34:00.024631 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014550 4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:34:00.024631 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014553 4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:34:00.024631 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014556 4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:34:00.024631 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014559 4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:34:00.024631 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014562 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:34:00.024631 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014565 4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:34:00.024631 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014569 4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:34:00.024631 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014572 4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:34:00.024631 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014575 4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:34:00.024631 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014578 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:34:00.024631 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014581 4531 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:34:00.024631 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014583 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:34:00.024631 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014586 4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:34:00.024631 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014589 4531 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:34:00.024631 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014592 4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:34:00.024631 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014595 4531 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:34:00.024631 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014597 4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:34:00.025737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014600 4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:34:00.025737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014602 4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:34:00.025737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014605 4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:34:00.025737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014607 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:34:00.025737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014610 4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:34:00.025737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014613 4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:34:00.025737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014616 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:34:00.025737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014620 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:34:00.025737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014623 4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:34:00.025737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014626 4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:34:00.025737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014628 4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:34:00.025737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014632 4531 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:34:00.025737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014634 4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:34:00.025737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014637 4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:34:00.025737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014640 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:34:00.025737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014642 4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:34:00.025737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014645 4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:34:00.025737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014647 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:34:00.025737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014650 4531 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:34:00.025737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014653 4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:34:00.026485 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014655 4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:34:00.026485 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014658 4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:34:00.026485 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014669 4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:34:00.026485 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014673 4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:34:00.026485 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014677 4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:34:00.026485 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014681 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:34:00.026485 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014684 4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:34:00.026485 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014687 4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:34:00.026485 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014689 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:34:00.026485 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014692 4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:34:00.026485 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014694 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:34:00.026485 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014697 4531 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:34:00.026485 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014699 4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:34:00.026485 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014704 4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:34:00.026485 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014708 4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:34:00.026485 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014711 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:34:00.026485 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014713 4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:34:00.026485 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014716 4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:34:00.026485 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014719 4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:34:00.027029 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014723 4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:34:00.027029 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014725 4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:34:00.027029 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014728 4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:34:00.027029 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014730 4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:34:00.027029 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014734 4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:34:00.027029 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014738 4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:34:00.027029 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014740 4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:34:00.027029 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014743 4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:34:00.027029 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014746 4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:34:00.027029 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014749 4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:34:00.027029 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014751 4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:34:00.027029 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014754 4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:34:00.027029 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014756 4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:34:00.027029 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014759 4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:34:00.027029 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014762 4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:34:00.027029 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014765 4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:34:00.027029 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014768 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:34:00.027029 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014771 4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:34:00.027029 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014773 4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:34:00.027029 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014776 4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:34:00.027717 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014778 4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:34:00.027717 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014781 4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:34:00.027717 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014783 4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:34:00.027717 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014786 4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:34:00.027717 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014788 4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:34:00.027717 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014791 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:34:00.027717 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014793 4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:34:00.027717 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014796 4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:34:00.027717 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014798 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:34:00.027717 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014801 4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:34:00.027717 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014803 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:34:00.027717 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014806 4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:34:00.027717 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014814 4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:34:00.027717 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014817 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:34:00.027717 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014819 4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:34:00.027717 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014823 4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:34:00.027717 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014827 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:34:00.027717 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014831 4531 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:34:00.027717 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014833 4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:34:00.028317 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014836 4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:34:00.028317 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014839 4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:34:00.028317 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014841 4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:34:00.028317 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014844 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:34:00.028317 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014846 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:34:00.028317 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014849 4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:34:00.028317 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014852 4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:34:00.028317 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014855 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:34:00.028317 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014858 4531 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:34:00.028317 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014860 4531 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:34:00.028317 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.014889 4531 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:34:00.028317 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014934 4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:34:00.028317 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014940 4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:34:00.028317 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014943 4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:34:00.028317 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014946 4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:34:00.028317 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014948 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:34:00.028714 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014951 4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:34:00.028714 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014955 4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:34:00.028714 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014957 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:34:00.028714 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014960 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:34:00.028714 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014962 4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:34:00.028714 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014965 4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:34:00.028714 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014968 4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:34:00.028714 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014970 4531 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:34:00.028714 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014973 4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:34:00.028714 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014977 4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:34:00.028714 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014979 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:34:00.028714 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014982 4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:34:00.028714 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014985 4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:34:00.028714 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014987 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:34:00.028714 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014991 4531 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:34:00.028714 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014994 4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:34:00.028714 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014997 4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:34:00.028714 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.014999 4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:34:00.028714 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015002 4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:34:00.028714 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015004 4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:34:00.029240 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015008 4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:34:00.029240 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015012 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:34:00.029240 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015015 4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:34:00.029240 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015018 4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:34:00.029240 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015021 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:34:00.029240 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015023 4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:34:00.029240 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015026 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:34:00.029240 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015029 4531 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:34:00.029240 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015032 4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:34:00.029240 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015036 4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:34:00.029240 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015039 4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:34:00.029240 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015042 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:34:00.029240 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015045 4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:34:00.029240 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015047 4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:34:00.029240 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015050 4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:34:00.029240 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015053 4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:34:00.029240 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015055 4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:34:00.029240 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015057 4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:34:00.029240 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015060 4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:34:00.029765 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015062 4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:34:00.029765 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015065 4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:34:00.029765 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015069 4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:34:00.029765 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015072 4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:34:00.029765 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015074 4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:34:00.029765 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015078 4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:34:00.029765 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015081 4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:34:00.029765 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015085 4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:34:00.029765 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015088 4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:34:00.029765 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015091 4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:34:00.029765 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015094 4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:34:00.029765 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015096 4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:34:00.029765 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015099 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:34:00.029765 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015101 4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:34:00.029765 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015104 4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:34:00.029765 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015107 4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:34:00.029765 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015109 4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:34:00.029765 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015112 4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:34:00.029765 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015114 4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:34:00.030372 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015117 4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:34:00.030372 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015119 4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:34:00.030372 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015140 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:34:00.030372 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015145 4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:34:00.030372 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015148 4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:34:00.030372 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015152 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:34:00.030372 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015156 4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:34:00.030372 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015160 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:34:00.030372 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015163 4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:34:00.030372 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015167 4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:34:00.030372 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015171 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:34:00.030372 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015176 4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:34:00.030372 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015180 4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:34:00.030372 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015183 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:34:00.030372 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015186 4531 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:34:00.030372 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015191 4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:34:00.030372 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015194 4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:34:00.030372 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015196 4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:34:00.030372 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015199 4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:34:00.030888 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015202 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:34:00.030888 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015206 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:34:00.030888 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015209 4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:34:00.030888 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015211 4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:34:00.030888 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015214 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:34:00.030888 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015216 4531 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:34:00.030888 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015219 4531 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:34:00.030888 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015222 4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:34:00.030888 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015225 4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:34:00.030888 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015228 4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:34:00.030888 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015230 4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:34:00.030888 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015233 4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:34:00.030888 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015235 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:34:00.030888 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015238 4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:34:00.030888 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015240 4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:34:00.030888 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015243 4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:34:00.030888 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015247 4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:34:00.030888 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015250 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:34:00.030888 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015252 4531 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:34:00.030888 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015256 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:34:00.031508 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015258 4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:34:00.031508 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015262 4531 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:34:00.031508 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015265 4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:34:00.031508 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.015268 4531 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:34:00.031508 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.019450 4531 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 18:34:00.031508 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.019474 4531 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 18:34:00.031508 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.019498 4531 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 18:34:00.031508 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.019503 4531 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 18:34:00.031508 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019566 4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:34:00.031508 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019573 4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:34:00.031508 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019577 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:34:00.031508 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019580 4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:34:00.031508 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019583 4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:34:00.031508 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019586 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:34:00.031508 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019588 4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:34:00.031508 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019593 4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:34:00.031508 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019598 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:34:00.031508 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019601 4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:34:00.031508 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019603 4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:34:00.031988 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019606 4531 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:34:00.031988 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019609 4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:34:00.031988 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019612 4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:34:00.031988 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019615 4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:34:00.031988 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019618 4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:34:00.031988 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019621 4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:34:00.031988 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019623 4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:34:00.031988 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019626 4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:34:00.031988 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019628 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:34:00.031988 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019631 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:34:00.031988 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019634 4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:34:00.031988 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019637 4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:34:00.031988 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019639 4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:34:00.031988 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019642 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:34:00.031988 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019645 4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:34:00.031988 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019647 4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:34:00.031988 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019650 4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:34:00.031988 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019653 4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:34:00.031988 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019659 4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:34:00.031988 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019662 4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:34:00.032517 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019665 4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:34:00.032517 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019667 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:34:00.032517 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019670 4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:34:00.032517 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019673 4531 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:34:00.032517 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019675 4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:34:00.032517 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019678 4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:34:00.032517 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019680 4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:34:00.032517 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019683 4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:34:00.032517 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019686 4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:34:00.032517 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019688 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:34:00.032517 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019691 4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:34:00.032517 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019693 4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:34:00.032517 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019696 4531 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:34:00.032517 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019698 4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:34:00.032517 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019701 4531 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:34:00.032517 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019704 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:34:00.032517 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019706 4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:34:00.032517 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019709 4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:34:00.032517 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019711 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:34:00.032517 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019714 4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:34:00.033063 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019716 4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:34:00.033063 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019719 4531 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:34:00.033063 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019721 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:34:00.033063 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019724 4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:34:00.033063 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019727 4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:34:00.033063 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019731 4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:34:00.033063 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019735 4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:34:00.033063 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019739 4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:34:00.033063 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019742 4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:34:00.033063 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019745 4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:34:00.033063 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019748 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:34:00.033063 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019751 4531 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:34:00.033063 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019754 4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:34:00.033063 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019756 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:34:00.033063 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019759 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:34:00.033063 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019762 4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:34:00.033063 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019764 4531 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:34:00.033063 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019768 4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:34:00.033063 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019770 4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:34:00.033737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019773 4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:34:00.033737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019776 4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:34:00.033737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019778 4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:34:00.033737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019781 4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:34:00.033737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019783 4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:34:00.033737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019786 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:34:00.033737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019789 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:34:00.033737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019791 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:34:00.033737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019794 4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:34:00.033737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019797 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:34:00.033737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019799 4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:34:00.033737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019802 4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:34:00.033737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019804 4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:34:00.033737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019807 4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:34:00.033737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019809 4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:34:00.033737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019812 4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:34:00.033737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019815 4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:34:00.033737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019818 4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:34:00.033737 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019820 4531 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:34:00.034337 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019823 4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:34:00.034337 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019826 4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:34:00.034337 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019828 4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:34:00.034337 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019832 4531 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:34:00.034337 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019835 4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:34:00.034337 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019838 4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:34:00.034337 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019841 4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:34:00.034337 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019844 4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:34:00.034337 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019847 4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:34:00.034337 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019849 4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:34:00.034337 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019852 4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:34:00.034337 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019855 4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:34:00.034337 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019858 4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:34:00.034337 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019860 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:34:00.034337 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019863 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:34:00.034337 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019866 4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:34:00.034337 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.019868 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:34:00.034875 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.019900 4531 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:34:00.034875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020049 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:34:00.034875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020054 4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:34:00.034875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020057 4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:34:00.034875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020060 4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:34:00.034875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020063 4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:34:00.034875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020066 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:34:00.034875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020068 4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:34:00.034875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020071 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:34:00.034875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020074 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:34:00.034875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020076 4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:34:00.034875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020080 4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:34:00.034875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020084 4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:34:00.034875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020088 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:34:00.034875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020090 4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:34:00.035465 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020093 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:34:00.035465 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020095 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:34:00.035465 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020098 4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:34:00.035465 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020101 4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:34:00.035465 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020104 4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:34:00.035465 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020106 4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:34:00.035465 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020109 4531 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:34:00.035465 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020112 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:34:00.035465 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020115 4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:34:00.035465 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020117 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:34:00.035465 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020120 4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:34:00.035465 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020122 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:34:00.035465 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020151 4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:34:00.035465 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020154 4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:34:00.035465 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020156 4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:34:00.035465 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020159 4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:34:00.035465 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020162 4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:34:00.035465 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020165 4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:34:00.035465 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020167 4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:34:00.036230 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020170 4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:34:00.036230 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020173 4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:34:00.036230 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020175 4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:34:00.036230 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020178 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:34:00.036230 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020181 4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:34:00.036230 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020183 4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:34:00.036230 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020186 4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:34:00.036230 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020189 4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:34:00.036230 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020191 4531 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:34:00.036230 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020194 4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:34:00.036230 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020197 4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:34:00.036230 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020199 4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:34:00.036230 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020202 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:34:00.036230 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020204 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:34:00.036230 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020207 4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:34:00.036230 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020209 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:34:00.036230 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020212 4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:34:00.036230 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020215 4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:34:00.036230 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020218 4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:34:00.036230 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020221 4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:34:00.036953 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020224 4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:34:00.036953 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020227 4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:34:00.036953 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020230 4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:34:00.036953 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020233 4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:34:00.036953 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020235 4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:34:00.036953 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020238 4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:34:00.036953 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020241 4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:34:00.036953 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020243 4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:34:00.036953 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020246 4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:34:00.036953 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020249 4531 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:34:00.036953 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020251 4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:34:00.036953 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020254 4531 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:34:00.036953 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020257 4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:34:00.036953 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020259 4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:34:00.036953 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020262 4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:34:00.036953 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020265 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:34:00.036953 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020267 4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:34:00.036953 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020270 4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:34:00.036953 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020272 4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:34:00.036953 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020275 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:34:00.037875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020277 4531 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:34:00.037875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020280 4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:34:00.037875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020282 4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:34:00.037875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020285 4531 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:34:00.037875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020287 4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:34:00.037875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020290 4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:34:00.037875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020293 4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:34:00.037875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020295 4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:34:00.037875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020298 4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:34:00.037875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020303 4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:34:00.037875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020306 4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:34:00.037875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020309 4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:34:00.037875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020312 4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:34:00.037875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020315 4531 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:34:00.037875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020318 4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:34:00.037875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020321 4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:34:00.037875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020324 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:34:00.037875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020326 4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:34:00.037875 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020329 4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:34:00.038471 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020331 4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:34:00.038471 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020334 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:34:00.038471 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020336 4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:34:00.038471 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020339 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:34:00.038471 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020341 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:34:00.038471 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020344 4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:34:00.038471 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020346 4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:34:00.038471 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020349 4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:34:00.038471 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020352 4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:34:00.038471 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020355 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:34:00.038471 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020357 4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:34:00.038471 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020360 4531 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:34:00.038471 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020363 4531 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:34:00.038471 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020366 4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:34:00.038471 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.020391 4531 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:34:00.038471 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020424 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:34:00.038871 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020429 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:34:00.038871 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020432 4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:34:00.038871 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020435 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:34:00.038871 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020441 4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:34:00.038871 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020444 4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:34:00.038871 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020447 4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:34:00.038871 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020450 4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:34:00.038871 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020452 4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:34:00.038871 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020455 4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:34:00.038871 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020457 4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:34:00.038871 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020460 4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:34:00.038871 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020462 4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:34:00.038871 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020465 4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:34:00.038871 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020468 4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:34:00.038871 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020471 4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:34:00.038871 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020474 4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:34:00.038871 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020476 4531 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:34:00.038871 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020479 4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:34:00.038871 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020482 4531 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:34:00.038871 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020484 4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:34:00.039671 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020487 4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:34:00.039671 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020489 4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:34:00.039671 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020492 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:34:00.039671 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020494 4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:34:00.039671 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020497 4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:34:00.039671 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020499 4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:34:00.039671 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020502 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:34:00.039671 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020504 4531 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:34:00.039671 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020507 4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:34:00.039671 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020509 4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:34:00.039671 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020512 4531 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:34:00.039671 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020514 4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:34:00.039671 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020517 4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:34:00.039671 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020520 4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:34:00.039671 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020522 4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:34:00.039671 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020527 4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:34:00.039671 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020531 4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:34:00.039671 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020536 4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:34:00.039671 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020539 4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:34:00.040452 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020542 4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:34:00.040452 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020545 4531 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:34:00.040452 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020548 4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:34:00.040452 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020551 4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:34:00.040452 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020554 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:34:00.040452 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020557 4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:34:00.040452 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020560 4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:34:00.040452 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020563 4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:34:00.040452 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020565 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:34:00.040452 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020568 4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:34:00.040452 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020570 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:34:00.040452 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020573 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:34:00.040452 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020575 4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:34:00.040452 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020579 4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:34:00.040452 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020581 4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:34:00.040452 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020584 4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:34:00.040452 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020586 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:34:00.040452 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020589 4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:34:00.040452 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020592 4531 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:34:00.040452 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020594 4531 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:34:00.041214 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020597 4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:34:00.041214 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020599 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:34:00.041214 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020603 4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:34:00.041214 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020606 4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:34:00.041214 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020609 4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:34:00.041214 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020611 4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:34:00.041214 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020614 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:34:00.041214 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020617 4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:34:00.041214 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020621 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:34:00.041214 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020624 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:34:00.041214 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020626 4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:34:00.041214 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020630 4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:34:00.041214 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020634 4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:34:00.041214 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020637 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:34:00.041214 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020640 4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:34:00.041214 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020643 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:34:00.041214 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020645 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:34:00.041214 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020648 4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:34:00.041214 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020651 4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:34:00.041996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020654 4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:34:00.041996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020657 4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:34:00.041996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020660 4531 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:34:00.041996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020662 4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:34:00.041996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020665 4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:34:00.041996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020667 4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:34:00.041996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020670 4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:34:00.041996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020672 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:34:00.041996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020675 4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:34:00.041996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020677 4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:34:00.041996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020680 4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:34:00.041996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020683 4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:34:00.041996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020685 4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:34:00.041996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020688 4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:34:00.041996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020690 4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:34:00.041996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020693 4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:34:00.041996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020695 4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:34:00.041996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020699 4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:34:00.041996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020701 4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:34:00.041996 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020704 4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:34:00.042815 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020707 4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:34:00.042815 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020711 4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:34:00.042815 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020714 4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:34:00.042815 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020717 4531 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:34:00.042815 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020719 4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:34:00.042815 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020722 4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:34:00.042815 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:00.020725 4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:34:00.042815 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.020798 4531 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 18:34:00.042815 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.021874 4531 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 18:34:00.042815 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.022140 4531 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 18:34:00.042815 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.022597 4531 server.go:1014] "Starting client certificate rotation" Oct 30 18:34:00.042815 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.023203 4531 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 18:34:00.042815 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.023504 4531 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:28:20 +0000 UTC" deadline="2028-03-25 14:00:02.678540428 +0000 UTC" Oct 30 18:34:00.042815 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.023547 4531 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12283h26m2.654995463s" Oct 30 18:34:00.042815 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.023900 4531 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:34:00.042815 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.025826 4531 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:34:00.043617 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.027595 4531 log.go:25] "Validated CRI v1 runtime API" Oct 30 18:34:00.043617 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.029459 4531 log.go:25] "Validated CRI v1 image API" Oct 30 18:34:00.043617 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.030328 4531 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 18:34:00.043617 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.032041 4531 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b1277cd2-d2fa-44bb-8a0d-3517cc2f9b60:/dev/nvme0n1p3 d18d8c20-68c9-4e92-98d0-3d37f1dbab67:/dev/nvme0n1p4] Oct 30 18:34:00.043794 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.032066 4531 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3df699b0fae271fb197e9b9b3b127585731af8cb1f09d00cb1a0b4368a49a505/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3df699b0fae271fb197e9b9b3b127585731af8cb1f09d00cb1a0b4368a49a505/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/44d18475766b206efd944f4e66107fafe7cc348b214cd44c7a889f8a58b44789/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/44d18475766b206efd944f4e66107fafe7cc348b214cd44c7a889f8a58b44789/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52378ab0946742667c5f06bbacbc5584f1402600bd3ed27f3d6cf257ec8643a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52378ab0946742667c5f06bbacbc5584f1402600bd3ed27f3d6cf257ec8643a1/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e5badf7e5aa75ed05a8c0fb739b661376477138c4bc5b96ac1d65f827d17426/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e5badf7e5aa75ed05a8c0fb739b661376477138c4bc5b96ac1d65f827d17426/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6bd1ea4b68f21ee96169e822f835d9eda8fa7cd8d83e3a2e701ea5778d166ee6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6bd1ea4b68f21ee96169e822f835d9eda8fa7cd8d83e3a2e701ea5778d166ee6/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/825ca0cf96a8bbb51b048ef97bf527945b170c242b223a8d2f73c006a5fb0c9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/825ca0cf96a8bbb51b048ef97bf527945b170c242b223a8d2f73c006a5fb0c9b/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/874e0ce10c321522a4ed6c5e132613dac0d85a8d2e24163cd160711c594f209b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/874e0ce10c321522a4ed6c5e132613dac0d85a8d2e24163cd160711c594f209b/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/879e92e7e28f8539194240f70c7e058330b43243540df67b153118eeab2bda0f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/879e92e7e28f8539194240f70c7e058330b43243540df67b153118eeab2bda0f/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aef8fe1f0bf8712e8a0975dfd9d83b032f9483336053dcd36ac79d96492ee3b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aef8fe1f0bf8712e8a0975dfd9d83b032f9483336053dcd36ac79d96492ee3b3/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b758f360492f15fcb6ca783e0055106b3c2b56d740b656e4fe8f3c47eafafae8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b758f360492f15fcb6ca783e0055106b3c2b56d740b656e4fe8f3c47eafafae8/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b76f3136135467860e65090a3b9d6d4f4bd4f5f84e6f6e4466c5e10e42af1155/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b76f3136135467860e65090a3b9d6d4f4bd4f5f84e6f6e4466c5e10e42af1155/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d3c2f17fd9744b24d93ee97a8a4df78541239379fbe521b0c90f98f33fc84c46/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d3c2f17fd9744b24d93ee97a8a4df78541239379fbe521b0c90f98f33fc84c46/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d719deca4e514317b4ed3d57beec0318ecf1d28ab3c74f9e38c3ad58e869af1b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d719deca4e514317b4ed3d57beec0318ecf1d28ab3c74f9e38c3ad58e869af1b/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df73aee8aeec5eb2ccaf7e6e156a0cd993c9e672c36dd57025e1966f698b1637/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df73aee8aeec5eb2ccaf7e6e156a0cd993c9e672c36dd57025e1966f698b1637/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee924fccdf6e416545bca8e22fe329ed989ffea1a1bd6e08a78736d3da4fa863/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee924fccdf6e416545bca8e22fe329ed989ffea1a1bd6e08a78736d3da4fa863/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f09c09059ed4098f4965821046cc526993b532bc4c4ada49da6a8f4c06ef1984/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f09c09059ed4098f4965821046cc526993b532bc4c4ada49da6a8f4c06ef1984/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/002053ca-649b-4a56-9ffd-25406d82fe99/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/002053ca-649b-4a56-9ffd-25406d82fe99/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02de5f04-abcb-440e-8e9e-55d41cd543e2/volumes/kubernetes.io~projected/kube-api-access-nmg24:{mountpoint:/var/lib/kubelet/pods/02de5f04-abcb-440e-8e9e-55d41cd543e2/volumes/kubernetes.io~projected/kube-api-access-nmg24 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10f6da88-0f08-47df-aa8f-7594b0dd3cc3/volumes/kubernetes.io~projected/kube-api-access-5rswb:{mountpoint:/var/lib/kubelet/pods/10f6da88-0f08-47df-aa8f-7594b0dd3cc3/volumes/kubernetes.io~projected/kube-api-access-5rswb major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17dae92a-ae2b-4c09-8001-48a81a8cf809/volumes/kubernetes.io~projected/kube-api-access-mxxl5:{mountpoint:/var/lib/kubelet/pods/17dae92a-ae2b-4c09-8001-48a81a8cf809/volumes/kubernetes.io~projected/kube-api-access-mxxl5 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2173c7a3-6b8f-4df7-8c17-cfd81932930a/volumes/kubernetes.io~projected/kube-api-access-lllm7:{mountpoint:/var/lib/kubelet/pods/2173c7a3-6b8f-4df7-8c17-cfd81932930a/volumes/kubernetes.io~projected/kube-api-access-lllm7 major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2173c7a3-6b8f-4df7-8c17-cfd81932930a/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/2173c7a3-6b8f-4df7-8c17-cfd81932930a/volumes/kubernetes.io~secret/signing-key major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d258432-7e24-4b20-a0e8-258aa1e1a4a5/volumes/kubernetes.io~projected/kube-api-access-2zdng:{mountpoint:/var/lib/kubelet/pods/2d258432-7e24-4b20-a0e8-258aa1e1a4a5/volumes/kubernetes.io~projected/kube-api-access-2zdng major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b5808a2-cbcf-4f2d-a401-191f429cc6be/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/3b5808a2-cbcf-4f2d-a401-191f429cc6be/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b5808a2-cbcf-4f2d-a401-191f429cc6be/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/3b5808a2-cbcf-4f2d-a401-191f429cc6be/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b5808a2-cbcf-4f2d-a401-191f429cc6be/volumes/kubernetes.io~projected/kube-api-access-zpw2g:{mountpoint:/var/lib/kubelet/pods/3b5808a2-cbcf-4f2d-a401-191f429cc6be/volumes/kubernetes.io~projected/kube-api-access-zpw2g major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/429722d3-91b1-4c92-b78e-19a12bc5bac9/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/429722d3-91b1-4c92-b78e-19a12bc5bac9/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ac7c1cf-9559-4281-be30-485e5e181d61/volumes/kubernetes.io~projected/kube-api-access-cmz4t:{mountpoint:/var/lib/kubelet/pods/4ac7c1cf-9559-4281-be30-485e5e181d61/volumes/kubernetes.io~projected/kube-api-access-cmz4t major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/886a6628-cf58-4784-85e0-d1ca14a2403c/volumes/kubernetes.io~projected/kube-api-access-xc7ct:{mountpoint:/var/lib/kubelet/pods/886a6628-cf58-4784-85e0-d1ca14a2403c/volumes/kubernetes.io~projected/kube-api-access-xc7ct major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/886a6628-cf58-4784-85e0-d1ca14a2403c/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/886a6628-cf58-4784-85e0-d1ca14a2403c/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af0639e4-bb41-4099-a9cf-03678c968c0b/volumes/kubernetes.io~projected/kube-api-access-9rfpj:{mountpoint:/var/lib/kubelet/pods/af0639e4-bb41-4099-a9cf-03678c968c0b/volumes/kubernetes.io~projected/kube-api-access-9rfpj major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1ea54d3-410e-48cf-80f6-46be7218f433/volumes/kubernetes.io~projected/kube-api-access-8r59q:{mountpoint:/var/lib/kubelet/pods/c1ea54d3-410e-48cf-80f6-46be7218f433/volumes/kubernetes.io~projected/kube-api-access-8r59q major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c84bbebc-7b6d-4e22-aacd-c98586717109/volumes/kubernetes.io~projected/kube-api-access-w2wzg:{mountpoint:/var/lib/kubelet/pods/c84bbebc-7b6d-4e22-aacd-c98586717109/volumes/kubernetes.io~projected/kube-api-access-w2wzg major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89/volumes/kubernetes.io~projected/kube-api-access-4qxnh:{mountpoint:/var/lib/kubelet/pods/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89/volumes/kubernetes.io~projected/kube-api-access-4qxnh major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d676d57c-8007-406d-8b45-e3a6c33ac3c9/volumes/kubernetes.io~projected/kube-api-access-cfbp8:{mountpoint:/var/lib/kubelet/pods/d676d57c-8007-406d-8b45-e3a6c33ac3c9/volumes/kubernetes.io~projected/kube-api-access-cfbp8 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9a334f2-12b2-4188-8bfa-7bc69559f024/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/e9a334f2-12b2-4188-8bfa-7bc69559f024/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9a334f2-12b2-4188-8bfa-7bc69559f024/volumes/kubernetes.io~projected/kube-api-access-mrf7w:{mountpoint:/var/lib/kubelet/pods/e9a334f2-12b2-4188-8bfa-7bc69559f024/volumes/kubernetes.io~projected/kube-api-access-mrf7w major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9a334f2-12b2-4188-8bfa-7bc69559f024/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/e9a334f2-12b2-4188-8bfa-7bc69559f024/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096/volumes/kubernetes.io~projected/kube-api-access-vrj48:{mountpoint:/var/lib/kubelet/pods/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096/volumes/kubernetes.io~projected/kube-api-access-vrj48 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eaa9cd18-eea8-4b24-b6af-c8cd6e278011/volumes/kubernetes.io~projected/kube-api-access-wpjjc:{mountpoint:/var/lib/kubelet/pods/eaa9cd18-eea8-4b24-b6af-c8cd6e278011/volumes/kubernetes.io~projected/kube-api-access-wpjjc major:0 minor:201 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/6441d7fe873ae67cf66145ff9a232dd4e177c919ccca22701ff184e6f1ec5080/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/8139d6b0488a80576d939ee40396396237bc7a02c748d5a4f09c32e93a165299/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/46702f81c21cbfb8321a41448cf9c0ec05b722f6f1a697b30019c4507bda523e/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/2750561154ae916989f819a6c569369da5a0d55d8b96a930b2e26445292a1364/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/661a7201413d0b2582048a8d0b7c21f031a609b1c660b33c285d6d2ad0b9674c/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/5308aee94d456be96e9d34cc2c37dab5bd75de1a914c74deac31f9a59311f9ae/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-121:{mountpoint:/var/lib/containers/storage/overlay/dfdf67f4555c725108b73a72a2a1ee1fbf741365a0fb02a79c5d44bc85d67c38/merged major:0 minor:121 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/f96fb9bf2dc0f0be0e9f834196ac5179bb8922a8a098f0ce811218062c3c1297/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/3403b1a38ff0289174ced488063a645f2a94607a28702cd58917c25b9b6aff91/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/d557be2df0c53189a0f648b0901bea39488c4647011f9dfbfcdb72856846193a/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/488ef7861433e18cf6c4ab75abc9fbfb46fb8ae60fea97b7f7337c1cf99e4b00/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/41bffc523ae72382e6eb6ae61b9d41b9ed48b5a0853a4ebf24dc330356b9dbe4/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/30e4e3ae5d8761ecc5a912f8a39e0b3bd3368e89e58037d28644b455fe9ea251/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/b8082a20865aa03b4341c39ebb5164e0f5d8d117ff672b822558b5a9bcc9f77e/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/39bab72103aa166936c16946acb8b48e8959c3fd1fda16df27a67ce8dcf11056/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/4de13cb1294edbac849180955dc54e4d7faf5e702f1627a7e9e3e6163907a4bd/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/9d3ff9b7840c8b4008560e80d6d8f58b458007276a7cf00a7879bd71a3063d8d/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/4040acc95159e4b9a70e08d66729be420dbee161dad1899fd215f88763d658bb/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/e4d4ccfc6c77f4238824066cfa2500a87a34c269b7bcbbba9f54216708370bf9/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/8d13b6a7f58c5f630b4ad6c24797a7c70df88024ecbd40eba70d3d3ccceb2ad5/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/a0041416a3a4fbab4409fc630c665bda9d8dd591927d1302d5a11e5b29eba42d/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/a794741c7859bd643e0d7bf56a18e484579e800d634c1f7a2e98fa3d6711fc8a/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/2cdc2a50ddfe5dc52856d99d57a5c10dddff5a95b70c490b802828316f8db077/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/636b6af667154940767cf1585b1d1260c47342c96e84913725e7502c092cfc01/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/6ad1c295450913fc664c9ce6eade6670b061fbaf900a46bdfbcca7424d461423/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/97362a3b1b49281502183ee6a7c8e70b4486dae4a012b4483c328e9da859e1d0/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/5dd7e924ccfc3f9892996ea3696106e4df5a9d40b0bd3f4bd4f51e49baaff1ce/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/ce81aa9097c0c3f8611ca2e5d1df87ddfa9e6133e1111f65872721d90907c34a/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/abd31d6034b5808c9fb8ec8bef468bde27fbac9b2f9cd88727a270f6bfc19aa1/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/6db8e246780f43d3d5b0f31ccab4515f7dda52c818c98473bfee4cacad7c560a/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/d878280868ed68a70f9d97862958f9e410f7682040c649b6f2a6730e89ef9736/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/076bf9b2fbd6298e9c5035d62a79871ab890fdaea6e4aecfe8afa90b10efc3ef/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/b1789661cb3e1dd5c54edb45a3cd0748baa6398a9dfcf85942ff5d8c7cdcadfb/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-65:{mountpoint:/var/lib/containers/storage/overlay/190b645305da91336adb8d01aa961023e31c9a976b9219c0d2ce189159449935/merged major:0 minor:65 fsType:overlay blockSize:0} overlay_0-67:{mountpoint:/var/lib/containers/storage/overlay/cd278a1516c5ac90d4adc4e289b70f07928a9d334fcfdb0fbec3502b9f84dbcc/merged major:0 minor:67 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/183009dd2ed4b9a840bc86e7dc6fe39f87aadc3b1650a28eabdb085358817e8d/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/2a380bacf000b4b5cdc3c9edbb32239dab1c56db54708af7f1bd7a5f66aad52d/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/23ce6d4959399fbc4a8580a68a6dccb82de9d3a66345b14f65c862318b9936cd/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/3428ac18bd369a85365efd4c70de4fca100b39ffeeabd50a862a4371414d02c2/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/89e2c2fb8c24a9f79bdf8da97df46010ec389037d4e24ae9891be20dfd28d085/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/4e6bb933b52ed0c6091acd630eb043bcaa0e68235f6cab12a5f9b749f864dd7c/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/19dfc083eed0a7ae8e85dd9daea6650269e762fd8893ff4a496224b8e3d522d1/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/915a6e0c3251cbfd72bfe34dafe1b45bfe68669a1b4b405bc5ddc28a502faf41/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 18:34:00.044037 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.038680 4531 manager.go:217] Machine: {Timestamp:2026-10-30 18:34:00.038194259 +0000 UTC m=+0.065299210 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3051213 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2770dd80d717336ce1649831bda449 SystemUUID:ec2770dd-80d7-1733-6ce1-649831bda449 BootID:3a6723dc-d773-4097-8f1c-bf2e00739c61 Filesystems:[{Device:/var/lib/kubelet/pods/e9a334f2-12b2-4188-8bfa-7bc69559f024/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/3b5808a2-cbcf-4f2d-a401-191f429cc6be/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eaa9cd18-eea8-4b24-b6af-c8cd6e278011/volumes/kubernetes.io~projected/kube-api-access-wpjjc DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/874e0ce10c321522a4ed6c5e132613dac0d85a8d2e24163cd160711c594f209b/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2173c7a3-6b8f-4df7-8c17-cfd81932930a/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:233 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f09c09059ed4098f4965821046cc526993b532bc4c4ada49da6a8f4c06ef1984/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c84bbebc-7b6d-4e22-aacd-c98586717109/volumes/kubernetes.io~projected/kube-api-access-w2wzg DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e9a334f2-12b2-4188-8bfa-7bc69559f024/volumes/kubernetes.io~projected/kube-api-access-mrf7w DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af0639e4-bb41-4099-a9cf-03678c968c0b/volumes/kubernetes.io~projected/kube-api-access-9rfpj DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2d258432-7e24-4b20-a0e8-258aa1e1a4a5/volumes/kubernetes.io~projected/kube-api-access-2zdng DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52378ab0946742667c5f06bbacbc5584f1402600bd3ed27f3d6cf257ec8643a1/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df73aee8aeec5eb2ccaf7e6e156a0cd993c9e672c36dd57025e1966f698b1637/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/825ca0cf96a8bbb51b048ef97bf527945b170c242b223a8d2f73c006a5fb0c9b/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-121 DeviceMajor:0 DeviceMinor:121 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/10f6da88-0f08-47df-aa8f-7594b0dd3cc3/volumes/kubernetes.io~projected/kube-api-access-5rswb DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d676d57c-8007-406d-8b45-e3a6c33ac3c9/volumes/kubernetes.io~projected/kube-api-access-cfbp8 DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/886a6628-cf58-4784-85e0-d1ca14a2403c/volumes/kubernetes.io~projected/kube-api-access-xc7ct DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89/volumes/kubernetes.io~projected/kube-api-access-4qxnh DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096/volumes/kubernetes.io~projected/kube-api-access-vrj48 DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee924fccdf6e416545bca8e22fe329ed989ffea1a1bd6e08a78736d3da4fa863/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-65 DeviceMajor:0 DeviceMinor:65 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b758f360492f15fcb6ca783e0055106b3c2b56d740b656e4fe8f3c47eafafae8/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aef8fe1f0bf8712e8a0975dfd9d83b032f9483336053dcd36ac79d96492ee3b3/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/002053ca-649b-4a56-9ffd-25406d82fe99/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d3c2f17fd9744b24d93ee97a8a4df78541239379fbe521b0c90f98f33fc84c46/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3df699b0fae271fb197e9b9b3b127585731af8cb1f09d00cb1a0b4368a49a505/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/886a6628-cf58-4784-85e0-d1ca14a2403c/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6bd1ea4b68f21ee96169e822f835d9eda8fa7cd8d83e3a2e701ea5778d166ee6/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/879e92e7e28f8539194240f70c7e058330b43243540df67b153118eeab2bda0f/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b5808a2-cbcf-4f2d-a401-191f429cc6be/volumes/kubernetes.io~projected/kube-api-access-zpw2g DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e5badf7e5aa75ed05a8c0fb739b661376477138c4bc5b96ac1d65f827d17426/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4ac7c1cf-9559-4281-be30-485e5e181d61/volumes/kubernetes.io~projected/kube-api-access-cmz4t DeviceMajor:0 DeviceMinor:222 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d719deca4e514317b4ed3d57beec0318ecf1d28ab3c74f9e38c3ad58e869af1b/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b5808a2-cbcf-4f2d-a401-191f429cc6be/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/429722d3-91b1-4c92-b78e-19a12bc5bac9/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/17dae92a-ae2b-4c09-8001-48a81a8cf809/volumes/kubernetes.io~projected/kube-api-access-mxxl5 DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1ea54d3-410e-48cf-80f6-46be7218f433/volumes/kubernetes.io~projected/kube-api-access-8r59q DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-67 DeviceMajor:0 DeviceMinor:67 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/44d18475766b206efd944f4e66107fafe7cc348b214cd44c7a889f8a58b44789/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2173c7a3-6b8f-4df7-8c17-cfd81932930a/volumes/kubernetes.io~projected/kube-api-access-lllm7 DeviceMajor:0 DeviceMinor:234 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b76f3136135467860e65090a3b9d6d4f4bd4f5f84e6f6e4466c5e10e42af1155/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9a334f2-12b2-4188-8bfa-7bc69559f024/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/02de5f04-abcb-440e-8e9e-55d41cd543e2/volumes/kubernetes.io~projected/kube-api-access-nmg24 DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:44d18475766b206 MacAddress:46:78:ca:a6:82:77 Speed:10000 Mtu:8901} {Name:874e0ce10c32152 MacAddress:16:e6:a3:0c:12:35 Speed:10000 Mtu:8901} {Name:aef8fe1f0bf8712 MacAddress:ca:9a:81:98:53:eb Speed:10000 Mtu:8901} {Name:b76f31361354678 MacAddress:ba:90:d7:0b:0a:88 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:2f:44:1d:f7:87 Speed:0 Mtu:9001} {Name:br-int MacAddress:3e:9f:98:ca:a1:43 Speed:0 Mtu:8901} {Name:d719deca4e51431 MacAddress:ea:d4:27:99:7b:71 Speed:10000 Mtu:8901} {Name:ee924fccdf6e416 MacAddress:72:2c:17:ca:2e:e8 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:2f:44:1d:f7:87 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:66:43:59:7d:5e:6a Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:32:24:72:ba:9d:a2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 18:34:00.044037 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.038970 4531 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 18:34:00.044037 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.039023 4531 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 18:34:00.044037 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.039491 4531 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 18:34:00.044037 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.039533 4531 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-56.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 18:34:00.044037 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.039765 4531 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 18:34:00.044037 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.039780 4531 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 18:34:00.044037 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.039801 4531 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:34:00.044037 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.039834 4531 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:34:00.044037 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.039996 4531 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 18:34:00.044037 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.040283 4531 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 18:34:00.044037 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.040437 4531 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 18:34:00.044037 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.040489 4531 kubelet.go:493] "Attempting to sync node with API server" Oct 30 18:34:00.044037 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.040511 4531 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 18:34:00.044037 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.040526 4531 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 18:34:00.044037 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.040542 4531 kubelet.go:405] "Adding apiserver pod source" Oct 30 18:34:00.044037 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.040554 4531 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 18:34:00.044037 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.042913 4531 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 18:34:00.044037 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.042976 4531 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:34:00.044037 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.043959 4531 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 18:34:00.048403 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.047346 4531 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 18:34:00.048403 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.047368 4531 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:34:00.048403 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.047509 4531 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 18:34:00.048403 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.047780 4531 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 18:34:00.048403 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.047821 4531 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 18:34:00.048403 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.047940 4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 18:34:00.048403 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.047956 4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 18:34:00.048403 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.047965 4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 18:34:00.048403 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.047976 4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 18:34:00.048403 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.048000 4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 18:34:00.048403 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.048009 4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 18:34:00.048403 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.048019 4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 18:34:00.048403 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.048026 4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 18:34:00.048403 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.048036 4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 18:34:00.048403 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.048045 4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 18:34:00.048403 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.048059 4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 18:34:00.048403 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.048074 4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 18:34:00.048403 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.048102 4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 18:34:00.048403 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.048110 4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 18:34:00.052241 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.052211 4531 server.go:1263] "Started kubelet" Oct 30 18:34:00.052402 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.052276 4531 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.052490 4531 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.052566 4531 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.053830 4531 server.go:317] "Adding debug handlers to kubelet server" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.053914 4531 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.056672 4531 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.057102 4531 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.057352 4531 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.057365 4531 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.057474 4531 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.059341 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eaa9cd18-eea8-4b24-b6af-c8cd6e278011" volumeName="kubernetes.io/empty-dir/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-tmp-dir" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.059388 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eaa9cd18-eea8-4b24-b6af-c8cd6e278011" volumeName="kubernetes.io/projected/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-kube-api-access-wpjjc" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.059403 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02de5f04-abcb-440e-8e9e-55d41cd543e2" volumeName="kubernetes.io/projected/02de5f04-abcb-440e-8e9e-55d41cd543e2-kube-api-access-nmg24" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.059417 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10f6da88-0f08-47df-aa8f-7594b0dd3cc3" volumeName="kubernetes.io/configmap/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-service-ca-bundle" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.059430 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10f6da88-0f08-47df-aa8f-7594b0dd3cc3" volumeName="kubernetes.io/empty-dir/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-archives-path" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.059442 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="429722d3-91b1-4c92-b78e-19a12bc5bac9" volumeName="kubernetes.io/secret/429722d3-91b1-4c92-b78e-19a12bc5bac9-original-pull-secret" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.059454 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af0639e4-bb41-4099-a9cf-03678c968c0b" volumeName="kubernetes.io/configmap/af0639e4-bb41-4099-a9cf-03678c968c0b-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.059467 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af0639e4-bb41-4099-a9cf-03678c968c0b" volumeName="kubernetes.io/empty-dir/af0639e4-bb41-4099-a9cf-03678c968c0b-data-volume" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.059482 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9a334f2-12b2-4188-8bfa-7bc69559f024" volumeName="kubernetes.io/configmap/e9a334f2-12b2-4188-8bfa-7bc69559f024-env-overrides" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.059495 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2173c7a3-6b8f-4df7-8c17-cfd81932930a" volumeName="kubernetes.io/configmap/2173c7a3-6b8f-4df7-8c17-cfd81932930a-signing-cabundle" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.059508 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2173c7a3-6b8f-4df7-8c17-cfd81932930a" volumeName="kubernetes.io/projected/2173c7a3-6b8f-4df7-8c17-cfd81932930a-kube-api-access-lllm7" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.059527 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b5808a2-cbcf-4f2d-a401-191f429cc6be" volumeName="kubernetes.io/empty-dir/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-tuned" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.059540 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02de5f04-abcb-440e-8e9e-55d41cd543e2" volumeName="kubernetes.io/configmap/02de5f04-abcb-440e-8e9e-55d41cd543e2-serviceca" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.059555 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10f6da88-0f08-47df-aa8f-7594b0dd3cc3" volumeName="kubernetes.io/configmap/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-trusted-ca-bundle" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.059570 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="886a6628-cf58-4784-85e0-d1ca14a2403c" volumeName="kubernetes.io/projected/886a6628-cf58-4784-85e0-d1ca14a2403c-kube-api-access-xc7ct" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.059585 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd3cee01-8466-456c-9b3d-c4f1fc5b4a89" volumeName="kubernetes.io/configmap/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.059597 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9a334f2-12b2-4188-8bfa-7bc69559f024" volumeName="kubernetes.io/configmap/e9a334f2-12b2-4188-8bfa-7bc69559f024-ovnkube-config" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.059612 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="002053ca-649b-4a56-9ffd-25406d82fe99" volumeName="kubernetes.io/secret/002053ca-649b-4a56-9ffd-25406d82fe99-agent-certs" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.059626 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2173c7a3-6b8f-4df7-8c17-cfd81932930a" volumeName="kubernetes.io/secret/2173c7a3-6b8f-4df7-8c17-cfd81932930a-signing-key" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.059637 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd3cee01-8466-456c-9b3d-c4f1fc5b4a89" volumeName="kubernetes.io/configmap/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.059648 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eaa9cd18-eea8-4b24-b6af-c8cd6e278011" volumeName="kubernetes.io/configmap/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-config-volume" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.059662 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="002053ca-649b-4a56-9ffd-25406d82fe99" volumeName="kubernetes.io/configmap/002053ca-649b-4a56-9ffd-25406d82fe99-konnectivity-ca" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.059677 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b5808a2-cbcf-4f2d-a401-191f429cc6be" volumeName="kubernetes.io/empty-dir/3b5808a2-cbcf-4f2d-a401-191f429cc6be-tmp" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.059689 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096" volumeName="kubernetes.io/projected/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-kube-api-access-vrj48" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.059703 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af0639e4-bb41-4099-a9cf-03678c968c0b" volumeName="kubernetes.io/projected/af0639e4-bb41-4099-a9cf-03678c968c0b-kube-api-access-9rfpj" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.060281 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1ea54d3-410e-48cf-80f6-46be7218f433" volumeName="kubernetes.io/projected/c1ea54d3-410e-48cf-80f6-46be7218f433-kube-api-access-8r59q" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.060301 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10f6da88-0f08-47df-aa8f-7594b0dd3cc3" volumeName="kubernetes.io/projected/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-kube-api-access-5rswb" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.060326 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17dae92a-ae2b-4c09-8001-48a81a8cf809" volumeName="kubernetes.io/projected/17dae92a-ae2b-4c09-8001-48a81a8cf809-kube-api-access-mxxl5" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.060343 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d258432-7e24-4b20-a0e8-258aa1e1a4a5" volumeName="kubernetes.io/empty-dir/2d258432-7e24-4b20-a0e8-258aa1e1a4a5-tmp-dir" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.060357 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d258432-7e24-4b20-a0e8-258aa1e1a4a5" volumeName="kubernetes.io/projected/2d258432-7e24-4b20-a0e8-258aa1e1a4a5-kube-api-access-2zdng" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.060369 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b5808a2-cbcf-4f2d-a401-191f429cc6be" volumeName="kubernetes.io/projected/3b5808a2-cbcf-4f2d-a401-191f429cc6be-kube-api-access-zpw2g" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.060401 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d676d57c-8007-406d-8b45-e3a6c33ac3c9" volumeName="kubernetes.io/configmap/d676d57c-8007-406d-8b45-e3a6c33ac3c9-multus-daemon-config" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.060413 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ac7c1cf-9559-4281-be30-485e5e181d61" volumeName="kubernetes.io/projected/4ac7c1cf-9559-4281-be30-485e5e181d61-kube-api-access-cmz4t" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.060433 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="886a6628-cf58-4784-85e0-d1ca14a2403c" volumeName="kubernetes.io/secret/886a6628-cf58-4784-85e0-d1ca14a2403c-cert" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.060443 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd3cee01-8466-456c-9b3d-c4f1fc5b4a89" volumeName="kubernetes.io/configmap/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-cni-binary-copy" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.060454 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd3cee01-8466-456c-9b3d-c4f1fc5b4a89" volumeName="kubernetes.io/projected/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-kube-api-access-4qxnh" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.060465 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d676d57c-8007-406d-8b45-e3a6c33ac3c9" volumeName="kubernetes.io/projected/d676d57c-8007-406d-8b45-e3a6c33ac3c9-kube-api-access-cfbp8" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.060476 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9a334f2-12b2-4188-8bfa-7bc69559f024" volumeName="kubernetes.io/projected/e9a334f2-12b2-4188-8bfa-7bc69559f024-kube-api-access-mrf7w" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.060488 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9a334f2-12b2-4188-8bfa-7bc69559f024" volumeName="kubernetes.io/secret/e9a334f2-12b2-4188-8bfa-7bc69559f024-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.060499 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1ea54d3-410e-48cf-80f6-46be7218f433" volumeName="kubernetes.io/configmap/c1ea54d3-410e-48cf-80f6-46be7218f433-iptables-alerter-script" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.060510 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c84bbebc-7b6d-4e22-aacd-c98586717109" volumeName="kubernetes.io/projected/c84bbebc-7b6d-4e22-aacd-c98586717109-kube-api-access-w2wzg" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.060525 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d676d57c-8007-406d-8b45-e3a6c33ac3c9" volumeName="kubernetes.io/configmap/d676d57c-8007-406d-8b45-e3a6c33ac3c9-cni-binary-copy" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.060537 4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9a334f2-12b2-4188-8bfa-7bc69559f024" volumeName="kubernetes.io/configmap/e9a334f2-12b2-4188-8bfa-7bc69559f024-ovnkube-script-lib" seLinuxMountContext="" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.060548 4531 reconstruct.go:97] "Volume reconstruction finished" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.060556 4531 reconciler.go:29] "Reconciler: start to sync state" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:00.061415 4531 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-56.us-west-1.compute.internal\" not found" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.061440 4531 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:28:21 +0000 UTC" deadline="2028-06-11 13:29:21.045090169 +0000 UTC" Oct 30 18:34:00.062696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.061455 4531 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14154h55m20.983637151s" Oct 30 18:34:00.053053 ip-10-0-133-56 systemd[1]: Started Kubernetes Kubelet. Oct 30 18:34:00.065360 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.062949 4531 factory.go:55] Registering systemd factory Oct 30 18:34:00.065360 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.062973 4531 factory.go:223] Registration of the systemd container factory successfully Oct 30 18:34:00.066208 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.065714 4531 factory.go:153] Registering CRI-O factory Oct 30 18:34:00.066208 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.065739 4531 factory.go:223] Registration of the crio container factory successfully Oct 30 18:34:00.066208 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.065804 4531 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 18:34:00.066208 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.065824 4531 factory.go:103] Registering Raw factory Oct 30 18:34:00.066208 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.065848 4531 manager.go:1199] Started watching for new ooms in manager Oct 30 18:34:00.066545 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.066345 4531 manager.go:319] Starting recovery of all containers Oct 30 18:34:00.089365 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.089304 4531 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 18:34:00.090931 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.090900 4531 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 18:34:00.091088 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.090959 4531 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 18:34:00.091088 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.090985 4531 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 18:34:00.091526 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:00.091032 4531 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 18:34:00.103517 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.103471 4531 generic.go:356] "Generic (PLEG): container finished" podID="76ae57dd60561f1d7425076e8b67e7e3" containerID="cd94820c78f9f788b6873f16f8768bc671b5eb566a22de9793ca9f0c13baa86e" exitCode=0 Oct 30 18:34:00.134968 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.134923 4531 generic.go:356] "Generic (PLEG): container finished" podID="cd3cee01-8466-456c-9b3d-c4f1fc5b4a89" containerID="d53eb76f3e3962af7b50fb95d4ba5128406718e08e8626b20585e972a1a61fec" exitCode=0 Oct 30 18:34:00.134968 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.134960 4531 generic.go:356] "Generic (PLEG): container finished" podID="cd3cee01-8466-456c-9b3d-c4f1fc5b4a89" containerID="349c62804450a51f6b8f9cc0a4dfb14b2380dc295110fac64b77647e654b416f" exitCode=0 Oct 30 18:34:00.134968 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.134973 4531 generic.go:356] "Generic (PLEG): container finished" podID="cd3cee01-8466-456c-9b3d-c4f1fc5b4a89" containerID="872122e5c58ec51c2cf2cf24b2665a6fffb802db24ceef797e1b084a48b4ef42" exitCode=0 Oct 30 18:34:00.135212 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.134982 4531 generic.go:356] "Generic (PLEG): container finished" podID="cd3cee01-8466-456c-9b3d-c4f1fc5b4a89" containerID="ed2d1d491dab8914fa34ddb3a832936ad46b658ea1fbaaf6952c9188a1ec3441" exitCode=0 Oct 30 18:34:00.135212 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.134990 4531 generic.go:356] "Generic (PLEG): container finished" podID="cd3cee01-8466-456c-9b3d-c4f1fc5b4a89" containerID="f978acb9238f8f2c5ccf020e519f220cab738b8c8946bfc396622cd62e1ceee5" exitCode=0 Oct 30 18:34:00.135212 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.134999 4531 generic.go:356] "Generic (PLEG): container finished" podID="cd3cee01-8466-456c-9b3d-c4f1fc5b4a89" containerID="0fde78119496ac1fc1a8ae3a24c28da592c05ada6f5002fe15da0ec12cb1b820" exitCode=0 Oct 30 18:34:00.143621 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.143593 4531 manager.go:324] Recovery completed Oct 30 18:34:00.161724 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:00.161566 4531 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-56.us-west-1.compute.internal\" not found" Oct 30 18:34:00.163757 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.163737 4531 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:34:00.168549 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.165178 4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:34:00.168549 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.165222 4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:34:00.168549 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.165237 4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 18:34:00.169794 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.169776 4531 cpu_manager.go:226] "Starting" policy="none" Oct 30 18:34:00.169794 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.169791 4531 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 18:34:00.169941 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.169796 4531 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:34:00.169941 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.169814 4531 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 18:34:00.170045 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.170018 4531 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 18:34:00.170103 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.170033 4531 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 18:34:00.170103 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.170060 4531 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 18:34:00.170103 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.170070 4531 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 18:34:00.170103 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.170078 4531 policy_none.go:50] "Start" Oct 30 18:34:00.170103 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.170091 4531 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 18:34:00.170103 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.170104 4531 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 18:34:00.170384 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.170316 4531 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 18:34:00.170384 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.170328 4531 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 18:34:00.170384 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.170336 4531 policy_none.go:44] "Start" Oct 30 18:34:00.170811 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.170790 4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:34:00.170872 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.170828 4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:34:00.170872 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.170842 4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 18:34:00.174656 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.174624 4531 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 18:34:00.177380 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.177360 4531 manager.go:341] "Starting Device Plugin manager" Oct 30 18:34:00.177480 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:00.177391 4531 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 18:34:00.177480 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.177402 4531 server.go:87] "Starting device plugin registration server" Oct 30 18:34:00.177716 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.177702 4531 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 18:34:00.177768 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.177717 4531 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 18:34:00.177829 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.177811 4531 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 18:34:00.177968 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.177922 4531 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 18:34:00.178042 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.177983 4531 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 18:34:00.178042 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.177991 4531 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 18:34:00.178147 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.178036 4531 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:34:00.177965709Z","UUID":"59e53b4f-f440-43d0-a8ec-ce5be9918c7a","Handler":null,"Name":"","Endpoint":""} Oct 30 18:34:00.179365 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.179346 4531 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 18:34:00.179443 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.179375 4531 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 18:34:00.179443 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:00.179348 4531 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 18:34:00.179525 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:00.179469 4531 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-56.us-west-1.compute.internal\" not found" Oct 30 18:34:00.192572 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.192046 4531 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal"] Oct 30 18:34:00.192723 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.192659 4531 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:34:00.193635 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.193618 4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:34:00.193744 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.193650 4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:34:00.193744 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.193667 4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 18:34:00.193932 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.193922 4531 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:34:00.194082 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.194047 4531 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:34:00.194695 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.194678 4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:34:00.194768 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.194703 4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:34:00.194768 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.194716 4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 18:34:00.194768 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.194731 4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:34:00.194768 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.194754 4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:34:00.194768 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.194767 4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 18:34:00.194917 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:00.194879 4531 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-56.us-west-1.compute.internal\" not found" node="ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:34:00.195009 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.194976 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal" event={"ID":"76ae57dd60561f1d7425076e8b67e7e3","Type":"ContainerStarted","Data":"dd0b66cf63f343b74fb38ce92507c5bc6bb98b5e94d3f3ef4a991378c7be7e70"} Oct 30 18:34:00.195043 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.195018 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal" event={"ID":"76ae57dd60561f1d7425076e8b67e7e3","Type":"ContainerDied","Data":"cd94820c78f9f788b6873f16f8768bc671b5eb566a22de9793ca9f0c13baa86e"} Oct 30 18:34:00.195043 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.195033 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal" event={"ID":"76ae57dd60561f1d7425076e8b67e7e3","Type":"ContainerStarted","Data":"b758f360492f15fcb6ca783e0055106b3c2b56d740b656e4fe8f3c47eafafae8"} Oct 30 18:34:00.195098 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.195046 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal" event={"ID":"d156aeb8e5a3750f522b65c4725c3f82","Type":"ContainerStarted","Data":"8517856f84378358cb894616c72fb46b303033f0309ca4c918656896e1032f03"} Oct 30 18:34:00.195098 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.195058 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal" event={"ID":"d156aeb8e5a3750f522b65c4725c3f82","Type":"ContainerStarted","Data":"52378ab0946742667c5f06bbacbc5584f1402600bd3ed27f3d6cf257ec8643a1"} Oct 30 18:34:00.195098 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.195086 4531 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:34:00.195846 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.195801 4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:34:00.195846 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.195835 4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:34:00.195989 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.195851 4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 18:34:00.196042 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:00.196017 4531 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-56.us-west-1.compute.internal\" not found" node="ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:34:00.255728 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.255697 4531 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:34:00.255728 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.255707 4531 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 18:34:00.278296 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.278256 4531 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:34:00.279407 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.279383 4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:34:00.279508 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.279421 4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:34:00.279508 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.279432 4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 18:34:00.279508 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.279466 4531 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:34:00.304706 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.304672 4531 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:34:00.305033 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.304677 4531 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:34:00.325967 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.325932 4531 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:34:00.356413 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.356375 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d156aeb8e5a3750f522b65c4725c3f82-config\") pod \"kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal\" (UID: \"d156aeb8e5a3750f522b65c4725c3f82\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:34:00.356668 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.356421 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76ae57dd60561f1d7425076e8b67e7e3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal\" (UID: \"76ae57dd60561f1d7425076e8b67e7e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:34:00.356668 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.356450 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76ae57dd60561f1d7425076e8b67e7e3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal\" (UID: \"76ae57dd60561f1d7425076e8b67e7e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:34:00.361753 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.361729 4531 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:34:00.406593 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.406557 4531 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:34:00.406780 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.406639 4531 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:34:00.407338 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.407316 4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeNotReady" Oct 30 18:34:00.407448 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.407356 4531 setters.go:546] "Node became not ready" node="ip-10-0-133-56.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T18:34:00Z","lastTransitionTime":"2026-10-30T18:34:00Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Oct 30 18:34:00.420828 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.420740 4531 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:34:00.420828 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:00.420805 4531 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:34:00.420828 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.420823 4531 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:34:00.440229 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.440189 4531 nodeinfomanager.go:471] existing CSINode "ip-10-0-133-56.us-west-1.compute.internal" is owned by different node (oldNodeID="29ea095f-5a42-44b2-80ac-e8a03257cd96", newNodeID=""), cleaning up... Oct 30 18:34:00.457402 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.457360 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76ae57dd60561f1d7425076e8b67e7e3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal\" (UID: \"76ae57dd60561f1d7425076e8b67e7e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:34:00.457499 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.457415 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d156aeb8e5a3750f522b65c4725c3f82-config\") pod \"kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal\" (UID: \"d156aeb8e5a3750f522b65c4725c3f82\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:34:00.457499 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.457481 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d156aeb8e5a3750f522b65c4725c3f82-config\") pod \"kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal\" (UID: \"d156aeb8e5a3750f522b65c4725c3f82\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:34:00.457596 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.457487 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76ae57dd60561f1d7425076e8b67e7e3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal\" (UID: \"76ae57dd60561f1d7425076e8b67e7e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:34:00.457596 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.457539 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76ae57dd60561f1d7425076e8b67e7e3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal\" (UID: \"76ae57dd60561f1d7425076e8b67e7e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:34:00.457596 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.457583 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76ae57dd60561f1d7425076e8b67e7e3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal\" (UID: \"76ae57dd60561f1d7425076e8b67e7e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:34:00.482417 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:00.482382 4531 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:34:00.482568 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:00.482440 4531 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:34:01.041939 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.041891 4531 apiserver.go:52] "Watching apiserver" Oct 30 18:34:01.137585 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.137551 4531 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:34:01.137770 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.137673 4531 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:34:01.164501 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.164459 4531 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 18:34:01.165039 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.165002 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-hh4r5","openshift-image-registry/node-ca-7c9gt","openshift-service-ca/service-ca-5fffc97bdf-7s7lj","openshift-dns/node-resolver-7jq9m","openshift-cluster-node-tuning-operator/tuned-996pn","openshift-kube-storage-version-migrator/migrator-66df5d496b-vd4zw","openshift-multus/multus-f2745","kube-system/kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal","openshift-insights/insights-runtime-extractor-v9jf7","openshift-dns/dns-default-49jb4","kube-system/global-pull-secret-syncer-frkss","openshift-insights/periodic-gathering-zhmhr-2v9dv","openshift-ovn-kubernetes/ovnkube-node-ptpzt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal","openshift-network-operator/iptables-alerter-qnqxv","openshift-multus/network-metrics-daemon-flgwv","openshift-ingress-canary/ingress-canary-hjrzq","openshift-multus/multus-additional-cni-plugins-nxpzg","kube-system/konnectivity-agent-sd88x","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh"] Oct 30 18:34:01.165562 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.165540 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:34:01.169303 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.169276 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-49jb4" Oct 30 18:34:01.169706 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.169689 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:34:01.197170 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.197120 4531 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:34:01.197331 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:01.197198 4531 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:34:01.198203 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.198186 4531 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:34:01.198260 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:01.198226 4531 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-56.us-west-1.compute.internal" Oct 30 18:34:01.220931 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.220895 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 18:34:01.221240 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.220935 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 18:34:01.221240 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.220908 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 18:34:01.221240 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.220895 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 18:34:01.221475 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.221270 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 18:34:01.221475 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.221353 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 18:34:01.221475 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.221362 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 18:34:01.221475 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.221374 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-g6mvb\"" Oct 30 18:34:01.221475 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.221380 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-b7gnv\"" Oct 30 18:34:01.221475 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.221389 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5qqcs\"" Oct 30 18:34:01.221475 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.221449 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qb49q\"" Oct 30 18:34:01.221475 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.221488 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cgz7p\"" Oct 30 18:34:01.221875 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.221552 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cr7s6\"" Oct 30 18:34:01.221875 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.221560 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-8wmd6\"" Oct 30 18:34:01.221875 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.221565 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wkpbb\"" Oct 30 18:34:01.221875 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.221561 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4bpx7\"" Oct 30 18:34:01.221875 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.221572 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 18:34:01.246861 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.246828 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 18:34:01.246861 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.246855 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 18:34:01.247093 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.246976 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 18:34:01.247093 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.247074 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 18:34:01.247254 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.247193 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 18:34:01.247305 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.247289 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 18:34:01.247483 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.247419 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 18:34:01.247751 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.247585 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 18:34:01.247751 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.247643 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 18:34:01.247751 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.247664 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 18:34:01.247751 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.247674 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 18:34:01.247751 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.247644 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 18:34:01.248290 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.247779 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 18:34:01.248290 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.247791 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 18:34:01.248290 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.247801 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 18:34:01.248290 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.247890 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 18:34:01.248290 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.248083 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 18:34:01.248290 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.248087 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 18:34:01.248290 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.248116 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:34:01.248290 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.248160 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 18:34:01.248290 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.248161 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:34:01.248290 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.248180 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 18:34:01.248290 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.248181 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 18:34:01.248290 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.248203 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 18:34:01.248290 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.248243 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 18:34:01.248290 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.248273 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 18:34:01.248826 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.248303 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 18:34:01.248826 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.248335 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 18:34:01.248826 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.248404 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 18:34:01.248826 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.248411 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 18:34:01.248826 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.248462 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 18:34:01.248826 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.248506 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 18:34:01.248826 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.248579 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 18:34:01.259610 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.259574 4531 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 18:34:01.264706 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.264682 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-sysctl-conf\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.264762 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.264716 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-cni-binary-copy\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:34:01.265239 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.264733 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/429722d3-91b1-4c92-b78e-19a12bc5bac9-dbus\") pod \"global-pull-secret-syncer-frkss\" (UID: \"429722d3-91b1-4c92-b78e-19a12bc5bac9\") " pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:34:01.265388 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.265364 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-modprobe-d\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.265520 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.265496 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-var-lib-kubelet\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.265586 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.265574 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc7ct\" (UniqueName: \"kubernetes.io/projected/886a6628-cf58-4784-85e0-d1ca14a2403c-kube-api-access-xc7ct\") pod \"ingress-canary-hjrzq\" (UID: \"886a6628-cf58-4784-85e0-d1ca14a2403c\") " pod="openshift-ingress-canary/ingress-canary-hjrzq" Oct 30 18:34:01.265796 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.265780 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/af0639e4-bb41-4099-a9cf-03678c968c0b-data-volume\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:34:01.265854 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.265832 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2173c7a3-6b8f-4df7-8c17-cfd81932930a-signing-key\") pod \"service-ca-5fffc97bdf-7s7lj\" (UID: \"2173c7a3-6b8f-4df7-8c17-cfd81932930a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7s7lj" Oct 30 18:34:01.265901 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.265866 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-cnibin\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.265954 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.265904 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:34:01.265954 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.265887 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-cni-binary-copy\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:34:01.266038 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.265966 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-slash\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.266038 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.266019 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-run-netns\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.266160 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.266051 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-run-ovn\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.266395 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.266377 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e9a334f2-12b2-4188-8bfa-7bc69559f024-env-overrides\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.266483 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.266468 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/02de5f04-abcb-440e-8e9e-55d41cd543e2-serviceca\") pod \"node-ca-7c9gt\" (UID: \"02de5f04-abcb-440e-8e9e-55d41cd543e2\") " pod="openshift-image-registry/node-ca-7c9gt" Oct 30 18:34:01.266529 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.266504 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-systemd\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.266578 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.266538 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpw2g\" (UniqueName: \"kubernetes.io/projected/3b5808a2-cbcf-4f2d-a401-191f429cc6be-kube-api-access-zpw2g\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.266627 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.266576 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:34:01.266726 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.266626 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/886a6628-cf58-4784-85e0-d1ca14a2403c-cert\") pod \"ingress-canary-hjrzq\" (UID: \"886a6628-cf58-4784-85e0-d1ca14a2403c\") " pod="openshift-ingress-canary/ingress-canary-hjrzq" Oct 30 18:34:01.266772 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.266728 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:34:01.266772 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.266754 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rfpj\" (UniqueName: \"kubernetes.io/projected/af0639e4-bb41-4099-a9cf-03678c968c0b-kube-api-access-9rfpj\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:34:01.266976 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.266940 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rswb\" (UniqueName: \"kubernetes.io/projected/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-kube-api-access-5rswb\") pod \"periodic-gathering-zhmhr-2v9dv\" (UID: \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\") " pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" Oct 30 18:34:01.267027 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.267004 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2173c7a3-6b8f-4df7-8c17-cfd81932930a-signing-cabundle\") pod \"service-ca-5fffc97bdf-7s7lj\" (UID: \"2173c7a3-6b8f-4df7-8c17-cfd81932930a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7s7lj" Oct 30 18:34:01.267076 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.267042 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:34:01.267122 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.267069 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d258432-7e24-4b20-a0e8-258aa1e1a4a5-tmp-dir\") pod \"node-resolver-7jq9m\" (UID: \"2d258432-7e24-4b20-a0e8-258aa1e1a4a5\") " pod="openshift-dns/node-resolver-7jq9m" Oct 30 18:34:01.267122 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.267108 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-var-lib-cni-bin\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.267241 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.267206 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-service-ca-bundle\") pod \"periodic-gathering-zhmhr-2v9dv\" (UID: \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\") " pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" Oct 30 18:34:01.267296 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.267272 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-var-lib-kubelet\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.267347 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.267310 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/af0639e4-bb41-4099-a9cf-03678c968c0b-data-volume\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:34:01.267426 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.267401 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:34:01.267474 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.267445 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qxnh\" (UniqueName: \"kubernetes.io/projected/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-kube-api-access-4qxnh\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:34:01.267526 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.267492 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-kubelet\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.267572 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.267523 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-node-log\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.267625 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.267579 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-log-socket\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.267671 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.267657 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1ea54d3-410e-48cf-80f6-46be7218f433-host-slash\") pod \"iptables-alerter-qnqxv\" (UID: \"c1ea54d3-410e-48cf-80f6-46be7218f433\") " pod="openshift-network-operator/iptables-alerter-qnqxv" Oct 30 18:34:01.267714 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.267692 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r59q\" (UniqueName: \"kubernetes.io/projected/c1ea54d3-410e-48cf-80f6-46be7218f433-kube-api-access-8r59q\") pod \"iptables-alerter-qnqxv\" (UID: \"c1ea54d3-410e-48cf-80f6-46be7218f433\") " pod="openshift-network-operator/iptables-alerter-qnqxv" Oct 30 18:34:01.267795 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.267766 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-config-volume\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:34:01.267864 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.267845 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2173c7a3-6b8f-4df7-8c17-cfd81932930a-signing-cabundle\") pod \"service-ca-5fffc97bdf-7s7lj\" (UID: \"2173c7a3-6b8f-4df7-8c17-cfd81932930a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7s7lj" Oct 30 18:34:01.267930 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.267916 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84bbebc-7b6d-4e22-aacd-c98586717109-metrics-certs\") pod \"network-metrics-daemon-flgwv\" (UID: \"c84bbebc-7b6d-4e22-aacd-c98586717109\") " pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:34:01.267981 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.267951 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-device-dir\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:34:01.268029 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.267969 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d258432-7e24-4b20-a0e8-258aa1e1a4a5-tmp-dir\") pod \"node-resolver-7jq9m\" (UID: \"2d258432-7e24-4b20-a0e8-258aa1e1a4a5\") " pod="openshift-dns/node-resolver-7jq9m" Oct 30 18:34:01.268075 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.268026 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-run-ovn-kubernetes\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.268556 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.268535 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-service-ca-bundle\") pod \"periodic-gathering-zhmhr-2v9dv\" (UID: \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\") " pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" Oct 30 18:34:01.268746 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.268730 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e9a334f2-12b2-4188-8bfa-7bc69559f024-env-overrides\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.269108 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.269089 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/02de5f04-abcb-440e-8e9e-55d41cd543e2-serviceca\") pod \"node-ca-7c9gt\" (UID: \"02de5f04-abcb-440e-8e9e-55d41cd543e2\") " pod="openshift-image-registry/node-ca-7c9gt" Oct 30 18:34:01.269191 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.269176 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.269243 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.269211 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-trusted-ca-bundle\") pod \"periodic-gathering-zhmhr-2v9dv\" (UID: \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\") " pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" Oct 30 18:34:01.269297 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.269246 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-etc-selinux\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:34:01.269297 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.269284 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/002053ca-649b-4a56-9ffd-25406d82fe99-agent-certs\") pod \"konnectivity-agent-sd88x\" (UID: \"002053ca-649b-4a56-9ffd-25406d82fe99\") " pod="kube-system/konnectivity-agent-sd88x" Oct 30 18:34:01.269401 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.269386 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e9a334f2-12b2-4188-8bfa-7bc69559f024-ovnkube-script-lib\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.269444 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.269416 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/429722d3-91b1-4c92-b78e-19a12bc5bac9-original-pull-secret\") pod \"global-pull-secret-syncer-frkss\" (UID: \"429722d3-91b1-4c92-b78e-19a12bc5bac9\") " pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:34:01.269491 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.269448 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:34:01.269491 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.269480 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-sys-fs\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:34:01.269589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.269531 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/002053ca-649b-4a56-9ffd-25406d82fe99-konnectivity-ca\") pod \"konnectivity-agent-sd88x\" (UID: \"002053ca-649b-4a56-9ffd-25406d82fe99\") " pod="kube-system/konnectivity-agent-sd88x" Oct 30 18:34:01.269589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.269563 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-run-k8s-cni-cncf-io\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.269683 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.269620 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-config-volume\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:34:01.269889 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.269858 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-var-lib-openvswitch\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.269889 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.269865 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:34:01.270049 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.269908 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-run-openvswitch\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.270049 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.269946 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e9a334f2-12b2-4188-8bfa-7bc69559f024-ovnkube-config\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.270049 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.269978 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-run\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.270049 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.270014 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrj48\" (UniqueName: \"kubernetes.io/projected/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-kube-api-access-vrj48\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:34:01.270269 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.270073 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-system-cni-dir\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.270269 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.270102 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d676d57c-8007-406d-8b45-e3a6c33ac3c9-multus-daemon-config\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.270269 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.270158 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/002053ca-649b-4a56-9ffd-25406d82fe99-konnectivity-ca\") pod \"konnectivity-agent-sd88x\" (UID: \"002053ca-649b-4a56-9ffd-25406d82fe99\") " pod="kube-system/konnectivity-agent-sd88x" Oct 30 18:34:01.270269 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.270164 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e9a334f2-12b2-4188-8bfa-7bc69559f024-ovnkube-script-lib\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.270269 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.270218 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-run-multus-certs\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.270269 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.270261 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-tuned\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.270537 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.270512 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d676d57c-8007-406d-8b45-e3a6c33ac3c9-cni-binary-copy\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.270591 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.270559 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-tuned\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.270639 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.270600 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e9a334f2-12b2-4188-8bfa-7bc69559f024-ovnkube-config\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.270639 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.270624 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-var-lib-cni-multus\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.270718 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.270633 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d676d57c-8007-406d-8b45-e3a6c33ac3c9-multus-daemon-config\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.270718 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.270701 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/002053ca-649b-4a56-9ffd-25406d82fe99-agent-certs\") pod \"konnectivity-agent-sd88x\" (UID: \"002053ca-649b-4a56-9ffd-25406d82fe99\") " pod="kube-system/konnectivity-agent-sd88x" Oct 30 18:34:01.270718 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.270709 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-run-systemd\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.270859 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.270776 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-cni-netd\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.270859 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.270816 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrf7w\" (UniqueName: \"kubernetes.io/projected/e9a334f2-12b2-4188-8bfa-7bc69559f024-kube-api-access-mrf7w\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.270961 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.270944 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d676d57c-8007-406d-8b45-e3a6c33ac3c9-cni-binary-copy\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.271029 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.271005 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/af0639e4-bb41-4099-a9cf-03678c968c0b-crio-socket\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:34:01.271252 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.271097 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-socket-dir\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:34:01.271252 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.271177 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-registration-dir\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:34:01.271556 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.271538 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-run-netns\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.271605 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.271588 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-multus-conf-dir\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.271669 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.271649 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-systemd-units\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.271737 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.271716 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af0639e4-bb41-4099-a9cf-03678c968c0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:34:01.271794 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.271771 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d258432-7e24-4b20-a0e8-258aa1e1a4a5-hosts-file\") pod \"node-resolver-7jq9m\" (UID: \"2d258432-7e24-4b20-a0e8-258aa1e1a4a5\") " pod="openshift-dns/node-resolver-7jq9m" Oct 30 18:34:01.271901 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.271885 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmz4t\" (UniqueName: \"kubernetes.io/projected/4ac7c1cf-9559-4281-be30-485e5e181d61-kube-api-access-cmz4t\") pod \"migrator-66df5d496b-vd4zw\" (UID: \"4ac7c1cf-9559-4281-be30-485e5e181d61\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vd4zw" Oct 30 18:34:01.271966 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.271930 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-multus-socket-dir-parent\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.271966 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.271949 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af0639e4-bb41-4099-a9cf-03678c968c0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:34:01.272101 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.271977 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:34:01.272101 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.272019 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-etc-openvswitch\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.272101 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.272059 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-cni-bin\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.272274 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.272114 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmg24\" (UniqueName: \"kubernetes.io/projected/02de5f04-abcb-440e-8e9e-55d41cd543e2-kube-api-access-nmg24\") pod \"node-ca-7c9gt\" (UID: \"02de5f04-abcb-440e-8e9e-55d41cd543e2\") " pod="openshift-image-registry/node-ca-7c9gt" Oct 30 18:34:01.272274 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.272169 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-kubernetes\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.272274 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.272267 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b5808a2-cbcf-4f2d-a401-191f429cc6be-tmp\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.272371 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.272304 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxxl5\" (UniqueName: \"kubernetes.io/projected/17dae92a-ae2b-4c09-8001-48a81a8cf809-kube-api-access-mxxl5\") pod \"network-check-target-hh4r5\" (UID: \"17dae92a-ae2b-4c09-8001-48a81a8cf809\") " pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:34:01.272409 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.272378 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-multus-cni-dir\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.272461 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.272429 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfbp8\" (UniqueName: \"kubernetes.io/projected/d676d57c-8007-406d-8b45-e3a6c33ac3c9-kube-api-access-cfbp8\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.272519 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.272475 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b5808a2-cbcf-4f2d-a401-191f429cc6be-tmp\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.272620 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.272597 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-tmp-dir\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:34:01.272697 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.272678 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-tmp-dir\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:34:01.272738 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.272703 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-sys\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.272738 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.272725 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-archives-path\") pod \"periodic-gathering-zhmhr-2v9dv\" (UID: \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\") " pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" Oct 30 18:34:01.272821 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.272780 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 18:34:01.272821 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.272787 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zdng\" (UniqueName: \"kubernetes.io/projected/2d258432-7e24-4b20-a0e8-258aa1e1a4a5-kube-api-access-2zdng\") pod \"node-resolver-7jq9m\" (UID: \"2d258432-7e24-4b20-a0e8-258aa1e1a4a5\") " pod="openshift-dns/node-resolver-7jq9m" Oct 30 18:34:01.272821 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.272791 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-archives-path\") pod \"periodic-gathering-zhmhr-2v9dv\" (UID: \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\") " pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" Oct 30 18:34:01.272968 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.272818 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-hostroot\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.272968 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.272861 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpjjc\" (UniqueName: \"kubernetes.io/projected/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-kube-api-access-wpjjc\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:34:01.272968 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.272894 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-lib-modules\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.272968 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.272964 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-host\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.273167 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.272985 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-etc-kubernetes\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.273167 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.273001 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-system-cni-dir\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:34:01.273167 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.273020 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1ea54d3-410e-48cf-80f6-46be7218f433-iptables-alerter-script\") pod \"iptables-alerter-qnqxv\" (UID: \"c1ea54d3-410e-48cf-80f6-46be7218f433\") " pod="openshift-network-operator/iptables-alerter-qnqxv" Oct 30 18:34:01.273167 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.273037 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lllm7\" (UniqueName: \"kubernetes.io/projected/2173c7a3-6b8f-4df7-8c17-cfd81932930a-kube-api-access-lllm7\") pod \"service-ca-5fffc97bdf-7s7lj\" (UID: \"2173c7a3-6b8f-4df7-8c17-cfd81932930a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7s7lj" Oct 30 18:34:01.273167 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.273078 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2wzg\" (UniqueName: \"kubernetes.io/projected/c84bbebc-7b6d-4e22-aacd-c98586717109-kube-api-access-w2wzg\") pod \"network-metrics-daemon-flgwv\" (UID: \"c84bbebc-7b6d-4e22-aacd-c98586717109\") " pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:34:01.273167 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.273148 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-cnibin\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:34:01.273357 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.273232 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e9a334f2-12b2-4188-8bfa-7bc69559f024-ovn-node-metrics-cert\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.273357 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.273280 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/429722d3-91b1-4c92-b78e-19a12bc5bac9-kubelet-config\") pod \"global-pull-secret-syncer-frkss\" (UID: \"429722d3-91b1-4c92-b78e-19a12bc5bac9\") " pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:34:01.273357 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.273313 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02de5f04-abcb-440e-8e9e-55d41cd543e2-host\") pod \"node-ca-7c9gt\" (UID: \"02de5f04-abcb-440e-8e9e-55d41cd543e2\") " pod="openshift-image-registry/node-ca-7c9gt" Oct 30 18:34:01.273357 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.273333 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1ea54d3-410e-48cf-80f6-46be7218f433-iptables-alerter-script\") pod \"iptables-alerter-qnqxv\" (UID: \"c1ea54d3-410e-48cf-80f6-46be7218f433\") " pod="openshift-network-operator/iptables-alerter-qnqxv" Oct 30 18:34:01.273357 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.273353 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-sysconfig\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.273493 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.273383 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-sysctl-d\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.281007 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.280984 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-trusted-ca-bundle\") pod \"periodic-gathering-zhmhr-2v9dv\" (UID: \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\") " pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" Oct 30 18:34:01.321593 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.321453 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 18:34:01.321946 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.321727 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 18:34:01.321946 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.321811 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 18:34:01.321946 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.321810 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 18:34:01.322232 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.321991 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k76xp\"" Oct 30 18:34:01.322232 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.321995 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-727z7\"" Oct 30 18:34:01.322232 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.321994 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9j6lr\"" Oct 30 18:34:01.322232 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.322044 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-sqz88\"" Oct 30 18:34:01.322232 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.322052 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 18:34:01.322232 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.322055 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jkxcr\"" Oct 30 18:34:01.322232 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.322073 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 18:34:01.322232 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.322081 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lwjtg\"" Oct 30 18:34:01.322232 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.322103 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lmrrc\"" Oct 30 18:34:01.322232 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.322098 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nh9hp\"" Oct 30 18:34:01.322232 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.322157 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-phb6k\"" Oct 30 18:34:01.324465 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.324444 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e9a334f2-12b2-4188-8bfa-7bc69559f024-ovn-node-metrics-cert\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.327155 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.327115 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rfpj\" (UniqueName: \"kubernetes.io/projected/af0639e4-bb41-4099-a9cf-03678c968c0b-kube-api-access-9rfpj\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:34:01.327155 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.327123 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r59q\" (UniqueName: \"kubernetes.io/projected/c1ea54d3-410e-48cf-80f6-46be7218f433-kube-api-access-8r59q\") pod \"iptables-alerter-qnqxv\" (UID: \"c1ea54d3-410e-48cf-80f6-46be7218f433\") " pod="openshift-network-operator/iptables-alerter-qnqxv" Oct 30 18:34:01.327314 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.327193 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qxnh\" (UniqueName: \"kubernetes.io/projected/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-kube-api-access-4qxnh\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:34:01.327370 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.327343 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpw2g\" (UniqueName: \"kubernetes.io/projected/3b5808a2-cbcf-4f2d-a401-191f429cc6be-kube-api-access-zpw2g\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.327419 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.327363 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2173c7a3-6b8f-4df7-8c17-cfd81932930a-signing-key\") pod \"service-ca-5fffc97bdf-7s7lj\" (UID: \"2173c7a3-6b8f-4df7-8c17-cfd81932930a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7s7lj" Oct 30 18:34:01.327419 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.327373 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rswb\" (UniqueName: \"kubernetes.io/projected/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-kube-api-access-5rswb\") pod \"periodic-gathering-zhmhr-2v9dv\" (UID: \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\") " pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" Oct 30 18:34:01.327484 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.327466 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc7ct\" (UniqueName: \"kubernetes.io/projected/886a6628-cf58-4784-85e0-d1ca14a2403c-kube-api-access-xc7ct\") pod \"ingress-canary-hjrzq\" (UID: \"886a6628-cf58-4784-85e0-d1ca14a2403c\") " pod="openshift-ingress-canary/ingress-canary-hjrzq" Oct 30 18:34:01.328423 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.328402 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrj48\" (UniqueName: \"kubernetes.io/projected/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-kube-api-access-vrj48\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:34:01.329290 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.329271 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/886a6628-cf58-4784-85e0-d1ca14a2403c-cert\") pod \"ingress-canary-hjrzq\" (UID: \"886a6628-cf58-4784-85e0-d1ca14a2403c\") " pod="openshift-ingress-canary/ingress-canary-hjrzq" Oct 30 18:34:01.330834 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.330811 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/429722d3-91b1-4c92-b78e-19a12bc5bac9-original-pull-secret\") pod \"global-pull-secret-syncer-frkss\" (UID: \"429722d3-91b1-4c92-b78e-19a12bc5bac9\") " pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:34:01.331084 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.331065 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmz4t\" (UniqueName: \"kubernetes.io/projected/4ac7c1cf-9559-4281-be30-485e5e181d61-kube-api-access-cmz4t\") pod \"migrator-66df5d496b-vd4zw\" (UID: \"4ac7c1cf-9559-4281-be30-485e5e181d61\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vd4zw" Oct 30 18:34:01.331156 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.331072 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrf7w\" (UniqueName: \"kubernetes.io/projected/e9a334f2-12b2-4188-8bfa-7bc69559f024-kube-api-access-mrf7w\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.336170 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.336113 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmg24\" (UniqueName: \"kubernetes.io/projected/02de5f04-abcb-440e-8e9e-55d41cd543e2-kube-api-access-nmg24\") pod \"node-ca-7c9gt\" (UID: \"02de5f04-abcb-440e-8e9e-55d41cd543e2\") " pod="openshift-image-registry/node-ca-7c9gt" Oct 30 18:34:01.336275 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.336180 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxxl5\" (UniqueName: \"kubernetes.io/projected/17dae92a-ae2b-4c09-8001-48a81a8cf809-kube-api-access-mxxl5\") pod \"network-check-target-hh4r5\" (UID: \"17dae92a-ae2b-4c09-8001-48a81a8cf809\") " pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:34:01.336771 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.336755 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfbp8\" (UniqueName: \"kubernetes.io/projected/d676d57c-8007-406d-8b45-e3a6c33ac3c9-kube-api-access-cfbp8\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.339920 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.339894 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpjjc\" (UniqueName: \"kubernetes.io/projected/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-kube-api-access-wpjjc\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:34:01.343254 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.343235 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zdng\" (UniqueName: \"kubernetes.io/projected/2d258432-7e24-4b20-a0e8-258aa1e1a4a5-kube-api-access-2zdng\") pod \"node-resolver-7jq9m\" (UID: \"2d258432-7e24-4b20-a0e8-258aa1e1a4a5\") " pod="openshift-dns/node-resolver-7jq9m" Oct 30 18:34:01.344751 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.344728 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2wzg\" (UniqueName: \"kubernetes.io/projected/c84bbebc-7b6d-4e22-aacd-c98586717109-kube-api-access-w2wzg\") pod \"network-metrics-daemon-flgwv\" (UID: \"c84bbebc-7b6d-4e22-aacd-c98586717109\") " pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:34:01.345282 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.345253 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lllm7\" (UniqueName: \"kubernetes.io/projected/2173c7a3-6b8f-4df7-8c17-cfd81932930a-kube-api-access-lllm7\") pod \"service-ca-5fffc97bdf-7s7lj\" (UID: \"2173c7a3-6b8f-4df7-8c17-cfd81932930a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7s7lj" Oct 30 18:34:01.346464 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.346443 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hjrzq_886a6628-cf58-4784-85e0-d1ca14a2403c/serve-healthcheck-canary/0.log" Oct 30 18:34:01.373828 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.373775 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84bbebc-7b6d-4e22-aacd-c98586717109-metrics-certs\") pod \"network-metrics-daemon-flgwv\" (UID: \"c84bbebc-7b6d-4e22-aacd-c98586717109\") " pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:34:01.373828 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.373825 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-device-dir\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:34:01.373994 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.373844 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-run-ovn-kubernetes\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.373994 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.373860 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.373994 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.373877 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-etc-selinux\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:34:01.373994 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.373894 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:34:01.373994 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.373908 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-sys-fs\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:34:01.373994 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.373939 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-run-ovn-kubernetes\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.373994 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.373968 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-run-k8s-cni-cncf-io\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.373994 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.373981 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-etc-selinux\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:34:01.374409 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374002 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-var-lib-openvswitch\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.374409 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.373970 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.374409 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374003 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-run-k8s-cni-cncf-io\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.374409 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:01.374010 4531 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 18:34:01.374409 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374038 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-run-openvswitch\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.374409 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.373939 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-device-dir\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:34:01.374409 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374034 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-var-lib-openvswitch\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.374409 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374041 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-sys-fs\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:34:01.374409 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374072 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-run\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.374409 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374096 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-run-openvswitch\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.374409 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374116 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-system-cni-dir\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.374409 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374120 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-run\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.374409 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374173 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-run-multus-certs\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.374409 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374196 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-system-cni-dir\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.374409 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374206 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-run-multus-certs\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.374409 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374212 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-var-lib-cni-multus\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.374409 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374240 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-run-systemd\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.374409 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374249 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-var-lib-cni-multus\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.374409 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:01.374275 4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls podName:eaa9cd18-eea8-4b24-b6af-c8cd6e278011 nodeName:}" failed. No retries permitted until 2026-10-30 18:34:01.874251385 +0000 UTC m=+1.901356343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls") pod "dns-default-49jb4" (UID: "eaa9cd18-eea8-4b24-b6af-c8cd6e278011") : secret "dns-default-metrics-tls" not found Oct 30 18:34:01.374409 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374278 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-run-systemd\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.374409 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374300 4531 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 18:34:01.374409 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374322 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-cni-netd\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.374409 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374378 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-cni-netd\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.374409 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374402 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/af0639e4-bb41-4099-a9cf-03678c968c0b-crio-socket\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:34:01.374409 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374425 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-socket-dir\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374456 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-registration-dir\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374492 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/af0639e4-bb41-4099-a9cf-03678c968c0b-crio-socket\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374509 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-socket-dir\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374517 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-registration-dir\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374532 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-run-netns\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374548 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-multus-conf-dir\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374562 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-systemd-units\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374564 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-run-netns\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374595 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-multus-conf-dir\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374601 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-systemd-units\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374620 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d258432-7e24-4b20-a0e8-258aa1e1a4a5-hosts-file\") pod \"node-resolver-7jq9m\" (UID: \"2d258432-7e24-4b20-a0e8-258aa1e1a4a5\") " pod="openshift-dns/node-resolver-7jq9m" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374642 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-multus-socket-dir-parent\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374692 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-multus-socket-dir-parent\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374692 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d258432-7e24-4b20-a0e8-258aa1e1a4a5-hosts-file\") pod \"node-resolver-7jq9m\" (UID: \"2d258432-7e24-4b20-a0e8-258aa1e1a4a5\") " pod="openshift-dns/node-resolver-7jq9m" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374711 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374732 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-etc-openvswitch\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374747 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-cni-bin\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374765 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-kubernetes\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374789 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-multus-cni-dir\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374792 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374810 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-etc-openvswitch\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374813 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-sys\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374841 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-kubernetes\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374845 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-hostroot\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374863 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-multus-cni-dir\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374879 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-hostroot\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374880 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-lib-modules\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374908 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-host\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374915 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-sys\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374930 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-etc-kubernetes\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374955 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-system-cni-dir\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374931 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-cni-bin\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374968 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-host\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374977 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-lib-modules\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374985 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-cnibin\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.374989 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-etc-kubernetes\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375010 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/429722d3-91b1-4c92-b78e-19a12bc5bac9-kubelet-config\") pod \"global-pull-secret-syncer-frkss\" (UID: \"429722d3-91b1-4c92-b78e-19a12bc5bac9\") " pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375016 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-system-cni-dir\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375036 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02de5f04-abcb-440e-8e9e-55d41cd543e2-host\") pod \"node-ca-7c9gt\" (UID: \"02de5f04-abcb-440e-8e9e-55d41cd543e2\") " pod="openshift-image-registry/node-ca-7c9gt" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375053 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd3cee01-8466-456c-9b3d-c4f1fc5b4a89-cnibin\") pod \"multus-additional-cni-plugins-nxpzg\" (UID: \"cd3cee01-8466-456c-9b3d-c4f1fc5b4a89\") " pod="openshift-multus/multus-additional-cni-plugins-nxpzg" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375058 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/429722d3-91b1-4c92-b78e-19a12bc5bac9-kubelet-config\") pod \"global-pull-secret-syncer-frkss\" (UID: \"429722d3-91b1-4c92-b78e-19a12bc5bac9\") " pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375070 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-sysconfig\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375095 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-sysctl-d\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375074 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02de5f04-abcb-440e-8e9e-55d41cd543e2-host\") pod \"node-ca-7c9gt\" (UID: \"02de5f04-abcb-440e-8e9e-55d41cd543e2\") " pod="openshift-image-registry/node-ca-7c9gt" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375120 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-sysctl-conf\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375174 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/429722d3-91b1-4c92-b78e-19a12bc5bac9-dbus\") pod \"global-pull-secret-syncer-frkss\" (UID: \"429722d3-91b1-4c92-b78e-19a12bc5bac9\") " pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375185 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-sysconfig\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375196 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-modprobe-d\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375222 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-var-lib-kubelet\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375240 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-sysctl-d\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375280 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-cnibin\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375284 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-var-lib-kubelet\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375288 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-modprobe-d\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375244 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-cnibin\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375305 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-sysctl-conf\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375316 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/429722d3-91b1-4c92-b78e-19a12bc5bac9-dbus\") pod \"global-pull-secret-syncer-frkss\" (UID: \"429722d3-91b1-4c92-b78e-19a12bc5bac9\") " pod="kube-system/global-pull-secret-syncer-frkss" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375330 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-slash\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375357 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-run-netns\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375379 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-run-ovn\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375386 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-slash\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375418 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-systemd\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375426 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-run-netns\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375444 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375463 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-run-ovn\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.375429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375475 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:34:01.378026 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375497 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3b5808a2-cbcf-4f2d-a401-191f429cc6be-etc-systemd\") pod \"tuned-996pn\" (UID: \"3b5808a2-cbcf-4f2d-a401-191f429cc6be\") " pod="openshift-cluster-node-tuning-operator/tuned-996pn" Oct 30 18:34:01.378026 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375508 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-var-lib-cni-bin\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.378026 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375534 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-var-lib-kubelet\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.378026 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375538 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wfpkh\" (UID: \"ea9d11f9-e0cb-40f4-ad8e-7925bdcdd096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wfpkh" Oct 30 18:34:01.378026 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:01.375553 4531 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 18:34:01.378026 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375562 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-kubelet\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.378026 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375566 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-var-lib-cni-bin\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.378026 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375584 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-node-log\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.378026 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375586 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d676d57c-8007-406d-8b45-e3a6c33ac3c9-host-var-lib-kubelet\") pod \"multus-f2745\" (UID: \"d676d57c-8007-406d-8b45-e3a6c33ac3c9\") " pod="openshift-multus/multus-f2745" Oct 30 18:34:01.378026 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:01.375597 4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls podName:af0639e4-bb41-4099-a9cf-03678c968c0b nodeName:}" failed. No retries permitted until 2026-10-30 18:34:01.875584264 +0000 UTC m=+1.902689220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9jf7" (UID: "af0639e4-bb41-4099-a9cf-03678c968c0b") : secret "insights-runtime-extractor-tls" not found Oct 30 18:34:01.378026 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375608 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-host-kubelet\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.378026 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375637 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-log-socket\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.378026 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375653 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1ea54d3-410e-48cf-80f6-46be7218f433-host-slash\") pod \"iptables-alerter-qnqxv\" (UID: \"c1ea54d3-410e-48cf-80f6-46be7218f433\") " pod="openshift-network-operator/iptables-alerter-qnqxv" Oct 30 18:34:01.378026 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375636 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-node-log\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.378026 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375706 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e9a334f2-12b2-4188-8bfa-7bc69559f024-log-socket\") pod \"ovnkube-node-ptpzt\" (UID: \"e9a334f2-12b2-4188-8bfa-7bc69559f024\") " pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:01.378026 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.375714 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1ea54d3-410e-48cf-80f6-46be7218f433-host-slash\") pod \"iptables-alerter-qnqxv\" (UID: \"c1ea54d3-410e-48cf-80f6-46be7218f433\") " pod="openshift-network-operator/iptables-alerter-qnqxv" Oct 30 18:34:01.378026 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.376704 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84bbebc-7b6d-4e22-aacd-c98586717109-metrics-certs\") pod \"network-metrics-daemon-flgwv\" (UID: \"c84bbebc-7b6d-4e22-aacd-c98586717109\") " pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:34:01.466479 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.466440 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flgwv" Oct 30 18:34:01.744588 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.744555 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-vd4zw_4ac7c1cf-9559-4281-be30-485e5e181d61/migrator/0.log" Oct 30 18:34:01.839758 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.839713 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-flgwv"] Oct 30 18:34:01.844285 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:01.844231 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84bbebc_7b6d_4e22_aacd_c98586717109.slice/crio-07e640fbf4de3864a0420443c14d016f1a105cb415975315358ba199224b6aa0 WatchSource:0}: Error finding container 07e640fbf4de3864a0420443c14d016f1a105cb415975315358ba199224b6aa0: Status 404 returned error can't find the container with id 07e640fbf4de3864a0420443c14d016f1a105cb415975315358ba199224b6aa0 Oct 30 18:34:01.846587 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.846570 4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:34:01.879365 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.879328 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:34:01.879365 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:01.879373 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:34:01.879569 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:01.879485 4531 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 18:34:01.879569 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:01.879492 4531 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 18:34:01.879569 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:01.879551 4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls podName:eaa9cd18-eea8-4b24-b6af-c8cd6e278011 nodeName:}" failed. No retries permitted until 2026-10-30 18:34:02.87953786 +0000 UTC m=+2.906642810 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls") pod "dns-default-49jb4" (UID: "eaa9cd18-eea8-4b24-b6af-c8cd6e278011") : secret "dns-default-metrics-tls" not found Oct 30 18:34:01.879569 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:01.879564 4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls podName:af0639e4-bb41-4099-a9cf-03678c968c0b nodeName:}" failed. No retries permitted until 2026-10-30 18:34:02.87955838 +0000 UTC m=+2.906663317 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9jf7" (UID: "af0639e4-bb41-4099-a9cf-03678c968c0b") : secret "insights-runtime-extractor-tls" not found Oct 30 18:34:02.141047 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:02.141003 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-flgwv" event={"ID":"c84bbebc-7b6d-4e22-aacd-c98586717109","Type":"ContainerStarted","Data":"07e640fbf4de3864a0420443c14d016f1a105cb415975315358ba199224b6aa0"} Oct 30 18:34:02.887659 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:02.887617 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:34:02.887659 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:02.887666 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:34:02.887895 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:02.887772 4531 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 18:34:02.887895 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:02.887787 4531 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 18:34:02.887895 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:02.887827 4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls podName:eaa9cd18-eea8-4b24-b6af-c8cd6e278011 nodeName:}" failed. No retries permitted until 2026-10-30 18:34:04.887814763 +0000 UTC m=+4.914919699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls") pod "dns-default-49jb4" (UID: "eaa9cd18-eea8-4b24-b6af-c8cd6e278011") : secret "dns-default-metrics-tls" not found Oct 30 18:34:02.887895 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:02.887852 4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls podName:af0639e4-bb41-4099-a9cf-03678c968c0b nodeName:}" failed. No retries permitted until 2026-10-30 18:34:04.88783504 +0000 UTC m=+4.914939997 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9jf7" (UID: "af0639e4-bb41-4099-a9cf-03678c968c0b") : secret "insights-runtime-extractor-tls" not found Oct 30 18:34:04.148876 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:04.148845 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-flgwv" event={"ID":"c84bbebc-7b6d-4e22-aacd-c98586717109","Type":"ContainerStarted","Data":"e8a596303b910d6076c2bf714cfcf6723c92a75971f123e66ae1267da4a2f835"} Oct 30 18:34:04.149293 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:04.148915 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-flgwv" event={"ID":"c84bbebc-7b6d-4e22-aacd-c98586717109","Type":"ContainerStarted","Data":"b55fca9de9405b5f169a275fa180d8956becec942535876a40d6b54149855594"} Oct 30 18:34:04.902144 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:04.902084 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:34:04.902317 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:04.902180 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:34:04.902317 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:04.902282 4531 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 18:34:04.902422 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:04.902339 4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls podName:af0639e4-bb41-4099-a9cf-03678c968c0b nodeName:}" failed. No retries permitted until 2026-10-30 18:34:08.902325092 +0000 UTC m=+8.929430043 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v9jf7" (UID: "af0639e4-bb41-4099-a9cf-03678c968c0b") : secret "insights-runtime-extractor-tls" not found Oct 30 18:34:04.904663 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:04.904641 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eaa9cd18-eea8-4b24-b6af-c8cd6e278011-metrics-tls\") pod \"dns-default-49jb4\" (UID: \"eaa9cd18-eea8-4b24-b6af-c8cd6e278011\") " pod="openshift-dns/dns-default-49jb4" Oct 30 18:34:05.070050 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:05.070010 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-49jb4" Oct 30 18:34:05.175501 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:05.175419 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:05.524825 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:05.524793 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-49jb4"] Oct 30 18:34:05.528214 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:05.528187 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa9cd18_eea8_4b24_b6af_c8cd6e278011.slice/crio-fe30261f2844968a7c3f683803c5e90841cb8a4283d242242d328b5cab1cae3c WatchSource:0}: Error finding container fe30261f2844968a7c3f683803c5e90841cb8a4283d242242d328b5cab1cae3c: Status 404 returned error can't find the container with id fe30261f2844968a7c3f683803c5e90841cb8a4283d242242d328b5cab1cae3c Oct 30 18:34:06.042215 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:06.042159 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:34:06.156947 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:06.156902 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-49jb4" event={"ID":"eaa9cd18-eea8-4b24-b6af-c8cd6e278011","Type":"ContainerStarted","Data":"fe30261f2844968a7c3f683803c5e90841cb8a4283d242242d328b5cab1cae3c"} Oct 30 18:34:08.933804 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:08.933755 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:34:08.936382 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:08.936339 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af0639e4-bb41-4099-a9cf-03678c968c0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v9jf7\" (UID: \"af0639e4-bb41-4099-a9cf-03678c968c0b\") " pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:34:08.970380 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:08.970338 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v9jf7" Oct 30 18:34:09.168427 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:09.168391 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-49jb4" event={"ID":"eaa9cd18-eea8-4b24-b6af-c8cd6e278011","Type":"ContainerStarted","Data":"cac72378774acd912f9de2b7caa1395286fece4732a5216c6c772711341ea7a4"} Oct 30 18:34:09.168578 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:09.168448 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-49jb4" event={"ID":"eaa9cd18-eea8-4b24-b6af-c8cd6e278011","Type":"ContainerStarted","Data":"eff662f61c3e32e73dd767f9b435821a606c3c7676ebfda4af50620737e88630"} Oct 30 18:34:09.168578 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:09.168551 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-49jb4" Oct 30 18:34:09.226500 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:09.226420 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:09.226668 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:09.226618 4531 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 18:34:09.451039 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:09.450991 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v9jf7"] Oct 30 18:34:09.456915 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:09.456875 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0639e4_bb41_4099_a9cf_03678c968c0b.slice/crio-7cb8bbb68d1f5cd6df1ba6ca93de789841c49d54bfdd14f0ec6505d3f26b606f WatchSource:0}: Error finding container 7cb8bbb68d1f5cd6df1ba6ca93de789841c49d54bfdd14f0ec6505d3f26b606f: Status 404 returned error can't find the container with id 7cb8bbb68d1f5cd6df1ba6ca93de789841c49d54bfdd14f0ec6505d3f26b606f Oct 30 18:34:10.173270 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:10.173230 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v9jf7" event={"ID":"af0639e4-bb41-4099-a9cf-03678c968c0b","Type":"ContainerStarted","Data":"43e158777d7da2499beea90f828750c197054c5311b5c59228a3dd7fecddcbe5"} Oct 30 18:34:10.173659 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:10.173296 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v9jf7" event={"ID":"af0639e4-bb41-4099-a9cf-03678c968c0b","Type":"ContainerStarted","Data":"7cb8bbb68d1f5cd6df1ba6ca93de789841c49d54bfdd14f0ec6505d3f26b606f"} Oct 30 18:34:10.804647 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:10.804614 4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-56.us-west-1.compute.internal" event="NodeReady" Oct 30 18:34:11.177989 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:11.177912 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v9jf7" event={"ID":"af0639e4-bb41-4099-a9cf-03678c968c0b","Type":"ContainerStarted","Data":"bcde9187ca7415c5f7a8884a83a47aec4d18659daca7b24a0d340ee594886921"} Oct 30 18:34:14.193839 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:14.193793 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v9jf7" event={"ID":"af0639e4-bb41-4099-a9cf-03678c968c0b","Type":"ContainerStarted","Data":"d586495a8d16363a74a1409e93b2d29af752bea6a3b9891c33ee7f2590d995da"} Oct 30 18:34:16.675116 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.675085 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwqfl"] Oct 30 18:34:16.696224 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.696195 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwqfl" Oct 30 18:34:16.792093 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.792049 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/620dbcf1-8778-481c-b395-82a607eab38f-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lwqfl\" (UID: \"620dbcf1-8778-481c-b395-82a607eab38f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwqfl" Oct 30 18:34:16.797876 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.797845 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-9tjgd\"" Oct 30 18:34:16.797876 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.797855 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 18:34:16.798857 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.798838 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-chsds"] Oct 30 18:34:16.815764 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.815725 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-chsds" Oct 30 18:34:16.848534 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.848499 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b4bd86cfb-cfpnq"] Oct 30 18:34:16.863245 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.863210 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwqfl"] Oct 30 18:34:16.863518 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.863259 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-chsds"] Oct 30 18:34:16.863518 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.863277 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b4bd86cfb-cfpnq"] Oct 30 18:34:16.863518 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.863347 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:16.892621 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.892588 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qstbv\" (UniqueName: \"kubernetes.io/projected/5b2d831b-06f5-4609-b791-abf171d60519-kube-api-access-qstbv\") pod \"downloads-6dd789c8b7-chsds\" (UID: \"5b2d831b-06f5-4609-b791-abf171d60519\") " pod="openshift-console/downloads-6dd789c8b7-chsds" Oct 30 18:34:16.892828 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.892642 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/620dbcf1-8778-481c-b395-82a607eab38f-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lwqfl\" (UID: \"620dbcf1-8778-481c-b395-82a607eab38f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwqfl" Oct 30 18:34:16.892828 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.892668 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b2d831b-06f5-4609-b791-abf171d60519-tmp\") pod \"downloads-6dd789c8b7-chsds\" (UID: \"5b2d831b-06f5-4609-b791-abf171d60519\") " pod="openshift-console/downloads-6dd789c8b7-chsds" Oct 30 18:34:16.895439 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.895414 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/620dbcf1-8778-481c-b395-82a607eab38f-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lwqfl\" (UID: \"620dbcf1-8778-481c-b395-82a607eab38f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwqfl" Oct 30 18:34:16.897635 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.897604 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 18:34:16.897635 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.897605 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 18:34:16.897753 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.897674 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-wkvt5\"" Oct 30 18:34:16.965889 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.965805 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 18:34:16.968934 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.968914 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 18:34:16.968934 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.968930 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 18:34:16.969084 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.968938 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-b9rv9\"" Oct 30 18:34:16.969084 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.968982 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 18:34:16.993926 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.993887 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qstbv\" (UniqueName: \"kubernetes.io/projected/5b2d831b-06f5-4609-b791-abf171d60519-kube-api-access-qstbv\") pod \"downloads-6dd789c8b7-chsds\" (UID: \"5b2d831b-06f5-4609-b791-abf171d60519\") " pod="openshift-console/downloads-6dd789c8b7-chsds" Oct 30 18:34:16.994189 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.993934 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65e878fb-34af-42a4-ab67-62b3ac06a947-image-registry-private-configuration\") pod \"image-registry-b4bd86cfb-cfpnq\" (UID: \"65e878fb-34af-42a4-ab67-62b3ac06a947\") " pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:16.994189 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.993960 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65e878fb-34af-42a4-ab67-62b3ac06a947-bound-sa-token\") pod \"image-registry-b4bd86cfb-cfpnq\" (UID: \"65e878fb-34af-42a4-ab67-62b3ac06a947\") " pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:16.994189 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.994077 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65e878fb-34af-42a4-ab67-62b3ac06a947-registry-tls\") pod \"image-registry-b4bd86cfb-cfpnq\" (UID: \"65e878fb-34af-42a4-ab67-62b3ac06a947\") " pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:16.994189 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.994183 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65e878fb-34af-42a4-ab67-62b3ac06a947-ca-trust-extracted\") pod \"image-registry-b4bd86cfb-cfpnq\" (UID: \"65e878fb-34af-42a4-ab67-62b3ac06a947\") " pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:16.994387 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.994215 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65e878fb-34af-42a4-ab67-62b3ac06a947-registry-certificates\") pod \"image-registry-b4bd86cfb-cfpnq\" (UID: \"65e878fb-34af-42a4-ab67-62b3ac06a947\") " pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:16.994387 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.994239 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65e878fb-34af-42a4-ab67-62b3ac06a947-trusted-ca\") pod \"image-registry-b4bd86cfb-cfpnq\" (UID: \"65e878fb-34af-42a4-ab67-62b3ac06a947\") " pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:16.994387 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.994291 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65e878fb-34af-42a4-ab67-62b3ac06a947-installation-pull-secrets\") pod \"image-registry-b4bd86cfb-cfpnq\" (UID: \"65e878fb-34af-42a4-ab67-62b3ac06a947\") " pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:16.994387 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.994345 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dhz8\" (UniqueName: \"kubernetes.io/projected/65e878fb-34af-42a4-ab67-62b3ac06a947-kube-api-access-8dhz8\") pod \"image-registry-b4bd86cfb-cfpnq\" (UID: \"65e878fb-34af-42a4-ab67-62b3ac06a947\") " pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:16.994387 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.994376 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b2d831b-06f5-4609-b791-abf171d60519-tmp\") pod \"downloads-6dd789c8b7-chsds\" (UID: \"5b2d831b-06f5-4609-b791-abf171d60519\") " pod="openshift-console/downloads-6dd789c8b7-chsds" Oct 30 18:34:16.994702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:16.994687 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b2d831b-06f5-4609-b791-abf171d60519-tmp\") pod \"downloads-6dd789c8b7-chsds\" (UID: \"5b2d831b-06f5-4609-b791-abf171d60519\") " pod="openshift-console/downloads-6dd789c8b7-chsds" Oct 30 18:34:17.019960 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:17.019922 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwqfl" Oct 30 18:34:17.055800 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:17.055775 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qstbv\" (UniqueName: \"kubernetes.io/projected/5b2d831b-06f5-4609-b791-abf171d60519-kube-api-access-qstbv\") pod \"downloads-6dd789c8b7-chsds\" (UID: \"5b2d831b-06f5-4609-b791-abf171d60519\") " pod="openshift-console/downloads-6dd789c8b7-chsds" Oct 30 18:34:17.095249 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:17.095213 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65e878fb-34af-42a4-ab67-62b3ac06a947-registry-tls\") pod \"image-registry-b4bd86cfb-cfpnq\" (UID: \"65e878fb-34af-42a4-ab67-62b3ac06a947\") " pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:17.095422 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:17.095259 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65e878fb-34af-42a4-ab67-62b3ac06a947-ca-trust-extracted\") pod \"image-registry-b4bd86cfb-cfpnq\" (UID: \"65e878fb-34af-42a4-ab67-62b3ac06a947\") " pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:17.095422 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:17.095279 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65e878fb-34af-42a4-ab67-62b3ac06a947-registry-certificates\") pod \"image-registry-b4bd86cfb-cfpnq\" (UID: \"65e878fb-34af-42a4-ab67-62b3ac06a947\") " pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:17.095422 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:17.095301 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65e878fb-34af-42a4-ab67-62b3ac06a947-trusted-ca\") pod \"image-registry-b4bd86cfb-cfpnq\" (UID: \"65e878fb-34af-42a4-ab67-62b3ac06a947\") " pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:17.095670 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:17.095652 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65e878fb-34af-42a4-ab67-62b3ac06a947-ca-trust-extracted\") pod \"image-registry-b4bd86cfb-cfpnq\" (UID: \"65e878fb-34af-42a4-ab67-62b3ac06a947\") " pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:17.095776 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:17.095760 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65e878fb-34af-42a4-ab67-62b3ac06a947-installation-pull-secrets\") pod \"image-registry-b4bd86cfb-cfpnq\" (UID: \"65e878fb-34af-42a4-ab67-62b3ac06a947\") " pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:17.095837 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:17.095787 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dhz8\" (UniqueName: \"kubernetes.io/projected/65e878fb-34af-42a4-ab67-62b3ac06a947-kube-api-access-8dhz8\") pod \"image-registry-b4bd86cfb-cfpnq\" (UID: \"65e878fb-34af-42a4-ab67-62b3ac06a947\") " pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:17.095886 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:17.095834 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65e878fb-34af-42a4-ab67-62b3ac06a947-image-registry-private-configuration\") pod \"image-registry-b4bd86cfb-cfpnq\" (UID: \"65e878fb-34af-42a4-ab67-62b3ac06a947\") " pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:17.095886 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:17.095856 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65e878fb-34af-42a4-ab67-62b3ac06a947-bound-sa-token\") pod \"image-registry-b4bd86cfb-cfpnq\" (UID: \"65e878fb-34af-42a4-ab67-62b3ac06a947\") " pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:17.097422 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:17.097392 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65e878fb-34af-42a4-ab67-62b3ac06a947-registry-certificates\") pod \"image-registry-b4bd86cfb-cfpnq\" (UID: \"65e878fb-34af-42a4-ab67-62b3ac06a947\") " pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:17.098040 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:17.097991 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65e878fb-34af-42a4-ab67-62b3ac06a947-trusted-ca\") pod \"image-registry-b4bd86cfb-cfpnq\" (UID: \"65e878fb-34af-42a4-ab67-62b3ac06a947\") " pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:17.102677 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:17.102651 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65e878fb-34af-42a4-ab67-62b3ac06a947-image-registry-private-configuration\") pod \"image-registry-b4bd86cfb-cfpnq\" (UID: \"65e878fb-34af-42a4-ab67-62b3ac06a947\") " pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:17.103033 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:17.103012 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65e878fb-34af-42a4-ab67-62b3ac06a947-installation-pull-secrets\") pod \"image-registry-b4bd86cfb-cfpnq\" (UID: \"65e878fb-34af-42a4-ab67-62b3ac06a947\") " pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:17.104897 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:17.104870 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65e878fb-34af-42a4-ab67-62b3ac06a947-registry-tls\") pod \"image-registry-b4bd86cfb-cfpnq\" (UID: \"65e878fb-34af-42a4-ab67-62b3ac06a947\") " pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:17.141403 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:17.141367 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-chsds" Oct 30 18:34:17.165955 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:17.165926 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65e878fb-34af-42a4-ab67-62b3ac06a947-bound-sa-token\") pod \"image-registry-b4bd86cfb-cfpnq\" (UID: \"65e878fb-34af-42a4-ab67-62b3ac06a947\") " pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:17.166155 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:17.165991 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dhz8\" (UniqueName: \"kubernetes.io/projected/65e878fb-34af-42a4-ab67-62b3ac06a947-kube-api-access-8dhz8\") pod \"image-registry-b4bd86cfb-cfpnq\" (UID: \"65e878fb-34af-42a4-ab67-62b3ac06a947\") " pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:17.186118 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:17.186084 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:17.402779 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:17.402738 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwqfl"] Oct 30 18:34:17.406580 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:17.406549 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620dbcf1_8778_481c_b395_82a607eab38f.slice/crio-2baaf40002b1444ef2c13d21ccbabc10eb3adf5656284d170c2d521436031269 WatchSource:0}: Error finding container 2baaf40002b1444ef2c13d21ccbabc10eb3adf5656284d170c2d521436031269: Status 404 returned error can't find the container with id 2baaf40002b1444ef2c13d21ccbabc10eb3adf5656284d170c2d521436031269 Oct 30 18:34:17.537963 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:17.537843 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-chsds"] Oct 30 18:34:17.541539 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:17.541509 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2d831b_06f5_4609_b791_abf171d60519.slice/crio-df1fd4827a4257350c326bac75c5fa3370a49f199a05525e051c2275cdaeb915 WatchSource:0}: Error finding container df1fd4827a4257350c326bac75c5fa3370a49f199a05525e051c2275cdaeb915: Status 404 returned error can't find the container with id df1fd4827a4257350c326bac75c5fa3370a49f199a05525e051c2275cdaeb915 Oct 30 18:34:17.593448 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:17.593411 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b4bd86cfb-cfpnq"] Oct 30 18:34:17.597988 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:17.597959 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e878fb_34af_42a4_ab67_62b3ac06a947.slice/crio-2a3cc86769468e294397e1697db5f673ea6b369280616380e6a86dd946436f1f WatchSource:0}: Error finding container 2a3cc86769468e294397e1697db5f673ea6b369280616380e6a86dd946436f1f: Status 404 returned error can't find the container with id 2a3cc86769468e294397e1697db5f673ea6b369280616380e6a86dd946436f1f Oct 30 18:34:18.208497 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:18.208458 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-chsds" event={"ID":"5b2d831b-06f5-4609-b791-abf171d60519","Type":"ContainerStarted","Data":"df1fd4827a4257350c326bac75c5fa3370a49f199a05525e051c2275cdaeb915"} Oct 30 18:34:18.210308 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:18.210275 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" event={"ID":"65e878fb-34af-42a4-ab67-62b3ac06a947","Type":"ContainerStarted","Data":"a8633467c01126c43ee2a4d5263fbc7b62208243cce49dc0024d2b981ab99473"} Oct 30 18:34:18.210470 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:18.210341 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" event={"ID":"65e878fb-34af-42a4-ab67-62b3ac06a947","Type":"ContainerStarted","Data":"2a3cc86769468e294397e1697db5f673ea6b369280616380e6a86dd946436f1f"} Oct 30 18:34:18.210470 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:18.210372 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:18.211586 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:18.211553 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwqfl" event={"ID":"620dbcf1-8778-481c-b395-82a607eab38f","Type":"ContainerStarted","Data":"2baaf40002b1444ef2c13d21ccbabc10eb3adf5656284d170c2d521436031269"} Oct 30 18:34:18.329860 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:18.329786 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" podStartSLOduration=2.329769556 podStartE2EDuration="2.329769556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:18.329411624 +0000 UTC m=+18.356516582" watchObservedRunningTime="2026-10-30 18:34:18.329769556 +0000 UTC m=+18.356874515" Oct 30 18:34:19.176289 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:19.176232 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-49jb4" Oct 30 18:34:20.221004 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:20.220966 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwqfl" event={"ID":"620dbcf1-8778-481c-b395-82a607eab38f","Type":"ContainerStarted","Data":"bf2daefdb3375845efde8f4e4429fd09ded455b60a0d3241acd3ab5223614a8a"} Oct 30 18:34:20.221550 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:20.221170 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwqfl" Oct 30 18:34:20.226838 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:20.226804 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwqfl" Oct 30 18:34:20.337299 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:20.337219 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwqfl" podStartSLOduration=2.285637619 podStartE2EDuration="4.33720062s" totalImagesPullingTime="2.051563001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:20.337012308 +0000 UTC m=+20.364117270" watchObservedRunningTime="2026-10-30 18:34:20.33720062 +0000 UTC m=+20.364305576" Oct 30 18:34:24.420508 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:24.420473 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sd88x" Oct 30 18:34:26.404779 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:26.404733 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:34:28.435107 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.435069 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tvdr4"] Oct 30 18:34:28.437564 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.437537 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.539811 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.539779 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 18:34:28.539811 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.539781 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 18:34:28.540067 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.539868 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tc629\"" Oct 30 18:34:28.582964 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.582929 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:34:28.582964 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.582954 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:34:28.583287 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.583002 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 18:34:28.590777 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.590730 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d36902b3-f9a4-4a16-b171-90e6eef10269-node-exporter-tls\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.590966 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.590851 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d36902b3-f9a4-4a16-b171-90e6eef10269-node-exporter-accelerators-collector-config\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.590966 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.590893 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d36902b3-f9a4-4a16-b171-90e6eef10269-node-exporter-wtmp\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.590966 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.590917 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d36902b3-f9a4-4a16-b171-90e6eef10269-node-exporter-textfile\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.590966 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.590954 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d36902b3-f9a4-4a16-b171-90e6eef10269-sys\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.591209 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.591062 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msj2p\" (UniqueName: \"kubernetes.io/projected/d36902b3-f9a4-4a16-b171-90e6eef10269-kube-api-access-msj2p\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.591209 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.591091 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d36902b3-f9a4-4a16-b171-90e6eef10269-metrics-client-ca\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.591209 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.591119 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d36902b3-f9a4-4a16-b171-90e6eef10269-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.591209 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.591174 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d36902b3-f9a4-4a16-b171-90e6eef10269-root\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.607587 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.607536 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 18:34:28.692444 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.692350 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d36902b3-f9a4-4a16-b171-90e6eef10269-sys\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.692444 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.692398 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msj2p\" (UniqueName: \"kubernetes.io/projected/d36902b3-f9a4-4a16-b171-90e6eef10269-kube-api-access-msj2p\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.692444 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.692426 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d36902b3-f9a4-4a16-b171-90e6eef10269-metrics-client-ca\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.692732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.692507 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d36902b3-f9a4-4a16-b171-90e6eef10269-sys\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.692732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.692570 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d36902b3-f9a4-4a16-b171-90e6eef10269-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.692732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.692613 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d36902b3-f9a4-4a16-b171-90e6eef10269-root\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.692732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.692647 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d36902b3-f9a4-4a16-b171-90e6eef10269-node-exporter-tls\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.692732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.692683 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d36902b3-f9a4-4a16-b171-90e6eef10269-node-exporter-accelerators-collector-config\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.692732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.692729 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d36902b3-f9a4-4a16-b171-90e6eef10269-node-exporter-wtmp\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.693027 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.692760 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d36902b3-f9a4-4a16-b171-90e6eef10269-node-exporter-textfile\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.693083 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.693061 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d36902b3-f9a4-4a16-b171-90e6eef10269-metrics-client-ca\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.693173 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.693094 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d36902b3-f9a4-4a16-b171-90e6eef10269-node-exporter-textfile\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.693229 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.693200 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d36902b3-f9a4-4a16-b171-90e6eef10269-root\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.693293 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.693249 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d36902b3-f9a4-4a16-b171-90e6eef10269-node-exporter-wtmp\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.693534 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.693508 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d36902b3-f9a4-4a16-b171-90e6eef10269-node-exporter-accelerators-collector-config\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.695541 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.695522 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d36902b3-f9a4-4a16-b171-90e6eef10269-node-exporter-tls\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.695649 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.695539 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d36902b3-f9a4-4a16-b171-90e6eef10269-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:28.763430 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:28.763396 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msj2p\" (UniqueName: \"kubernetes.io/projected/d36902b3-f9a4-4a16-b171-90e6eef10269-kube-api-access-msj2p\") pod \"node-exporter-tvdr4\" (UID: \"d36902b3-f9a4-4a16-b171-90e6eef10269\") " pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:29.052565 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:29.052512 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tvdr4" Oct 30 18:34:29.066179 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:29.066110 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36902b3_f9a4_4a16_b171_90e6eef10269.slice/crio-feae1ffc1c51d7d5395b4038b62b58081016cf8d99570da0a73b3b4c19560f89 WatchSource:0}: Error finding container feae1ffc1c51d7d5395b4038b62b58081016cf8d99570da0a73b3b4c19560f89: Status 404 returned error can't find the container with id feae1ffc1c51d7d5395b4038b62b58081016cf8d99570da0a73b3b4c19560f89 Oct 30 18:34:29.081778 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:29.081743 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ptpzt" Oct 30 18:34:29.254695 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:29.254659 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tvdr4" event={"ID":"d36902b3-f9a4-4a16-b171-90e6eef10269","Type":"ContainerStarted","Data":"feae1ffc1c51d7d5395b4038b62b58081016cf8d99570da0a73b3b4c19560f89"} Oct 30 18:34:31.263953 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:31.263912 4531 generic.go:356] "Generic (PLEG): container finished" podID="d36902b3-f9a4-4a16-b171-90e6eef10269" containerID="97f30144d1f3bfd3e193f07c452403c3e3f442ea0467a5e1558b92ee9efc6a61" exitCode=0 Oct 30 18:34:31.264431 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:31.263963 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tvdr4" event={"ID":"d36902b3-f9a4-4a16-b171-90e6eef10269","Type":"ContainerDied","Data":"97f30144d1f3bfd3e193f07c452403c3e3f442ea0467a5e1558b92ee9efc6a61"} Oct 30 18:34:32.270750 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:32.270712 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tvdr4" event={"ID":"d36902b3-f9a4-4a16-b171-90e6eef10269","Type":"ContainerStarted","Data":"ae68a181f169c27de121bdec49f81f817529ddf4c8267f6fbae05a075bbd8663"} Oct 30 18:34:32.271252 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:32.270768 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tvdr4" event={"ID":"d36902b3-f9a4-4a16-b171-90e6eef10269","Type":"ContainerStarted","Data":"0d83eefe3a91fd11049978bf4b1fa17dcdf761623085837ee0b403b2d961b2e1"} Oct 30 18:34:32.389252 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:32.389170 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tvdr4" podStartSLOduration=2.951152294 podStartE2EDuration="4.389150968s" totalImagesPullingTime="1.352878431s" totalInitContainerRuntime="85.120243ms" isStatefulPod=true podCreationTimestamp="2026-10-30 18:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:32.388529236 +0000 UTC m=+32.415634198" watchObservedRunningTime="2026-10-30 18:34:32.389150968 +0000 UTC m=+32.416255918" Oct 30 18:34:34.221087 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.221049 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5795d67875-qlc84"] Oct 30 18:34:34.226411 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.226382 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" Oct 30 18:34:34.249220 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.249187 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5795d67875-qlc84"] Oct 30 18:34:34.279431 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.279329 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-chsds" event={"ID":"5b2d831b-06f5-4609-b791-abf171d60519","Type":"ContainerStarted","Data":"575c7f159d7b099a49daa74a27f549462422813475fe9c0792b02c28f56726ba"} Oct 30 18:34:34.279644 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.279603 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-chsds" Oct 30 18:34:34.288314 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.287673 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 18:34:34.288314 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.287699 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 18:34:34.288314 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.287709 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 18:34:34.288314 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.287700 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-2t8ft\"" Oct 30 18:34:34.293944 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.293917 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-chsds" Oct 30 18:34:34.297428 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.297381 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 18:34:34.297428 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.297389 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 18:34:34.338683 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.338642 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/969e9a2a-59b9-48fd-978a-9027833aeb8d-federate-client-tls\") pod \"telemeter-client-5795d67875-qlc84\" (UID: \"969e9a2a-59b9-48fd-978a-9027833aeb8d\") " pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" Oct 30 18:34:34.338869 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.338698 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzfln\" (UniqueName: \"kubernetes.io/projected/969e9a2a-59b9-48fd-978a-9027833aeb8d-kube-api-access-wzfln\") pod \"telemeter-client-5795d67875-qlc84\" (UID: \"969e9a2a-59b9-48fd-978a-9027833aeb8d\") " pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" Oct 30 18:34:34.338869 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.338737 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/969e9a2a-59b9-48fd-978a-9027833aeb8d-telemeter-client-tls\") pod \"telemeter-client-5795d67875-qlc84\" (UID: \"969e9a2a-59b9-48fd-978a-9027833aeb8d\") " pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" Oct 30 18:34:34.338869 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.338804 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/969e9a2a-59b9-48fd-978a-9027833aeb8d-serving-certs-ca-bundle\") pod \"telemeter-client-5795d67875-qlc84\" (UID: \"969e9a2a-59b9-48fd-978a-9027833aeb8d\") " pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" Oct 30 18:34:34.338869 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.338860 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/969e9a2a-59b9-48fd-978a-9027833aeb8d-metrics-client-ca\") pod \"telemeter-client-5795d67875-qlc84\" (UID: \"969e9a2a-59b9-48fd-978a-9027833aeb8d\") " pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" Oct 30 18:34:34.339096 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.338943 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/969e9a2a-59b9-48fd-978a-9027833aeb8d-secret-telemeter-client\") pod \"telemeter-client-5795d67875-qlc84\" (UID: \"969e9a2a-59b9-48fd-978a-9027833aeb8d\") " pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" Oct 30 18:34:34.339096 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.338996 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/969e9a2a-59b9-48fd-978a-9027833aeb8d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5795d67875-qlc84\" (UID: \"969e9a2a-59b9-48fd-978a-9027833aeb8d\") " pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" Oct 30 18:34:34.339096 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.339054 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/969e9a2a-59b9-48fd-978a-9027833aeb8d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5795d67875-qlc84\" (UID: \"969e9a2a-59b9-48fd-978a-9027833aeb8d\") " pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" Oct 30 18:34:34.378982 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.378946 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 18:34:34.439684 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.439637 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/969e9a2a-59b9-48fd-978a-9027833aeb8d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5795d67875-qlc84\" (UID: \"969e9a2a-59b9-48fd-978a-9027833aeb8d\") " pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" Oct 30 18:34:34.439892 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.439707 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/969e9a2a-59b9-48fd-978a-9027833aeb8d-federate-client-tls\") pod \"telemeter-client-5795d67875-qlc84\" (UID: \"969e9a2a-59b9-48fd-978a-9027833aeb8d\") " pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" Oct 30 18:34:34.439892 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.439740 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzfln\" (UniqueName: \"kubernetes.io/projected/969e9a2a-59b9-48fd-978a-9027833aeb8d-kube-api-access-wzfln\") pod \"telemeter-client-5795d67875-qlc84\" (UID: \"969e9a2a-59b9-48fd-978a-9027833aeb8d\") " pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" Oct 30 18:34:34.439892 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.439782 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/969e9a2a-59b9-48fd-978a-9027833aeb8d-telemeter-client-tls\") pod \"telemeter-client-5795d67875-qlc84\" (UID: \"969e9a2a-59b9-48fd-978a-9027833aeb8d\") " pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" Oct 30 18:34:34.439892 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.439838 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/969e9a2a-59b9-48fd-978a-9027833aeb8d-serving-certs-ca-bundle\") pod \"telemeter-client-5795d67875-qlc84\" (UID: \"969e9a2a-59b9-48fd-978a-9027833aeb8d\") " pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" Oct 30 18:34:34.439892 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.439860 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/969e9a2a-59b9-48fd-978a-9027833aeb8d-metrics-client-ca\") pod \"telemeter-client-5795d67875-qlc84\" (UID: \"969e9a2a-59b9-48fd-978a-9027833aeb8d\") " pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" Oct 30 18:34:34.440161 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.439893 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/969e9a2a-59b9-48fd-978a-9027833aeb8d-secret-telemeter-client\") pod \"telemeter-client-5795d67875-qlc84\" (UID: \"969e9a2a-59b9-48fd-978a-9027833aeb8d\") " pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" Oct 30 18:34:34.440161 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.439930 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/969e9a2a-59b9-48fd-978a-9027833aeb8d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5795d67875-qlc84\" (UID: \"969e9a2a-59b9-48fd-978a-9027833aeb8d\") " pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" Oct 30 18:34:34.441070 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.440724 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/969e9a2a-59b9-48fd-978a-9027833aeb8d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5795d67875-qlc84\" (UID: \"969e9a2a-59b9-48fd-978a-9027833aeb8d\") " pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" Oct 30 18:34:34.441222 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.441187 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/969e9a2a-59b9-48fd-978a-9027833aeb8d-metrics-client-ca\") pod \"telemeter-client-5795d67875-qlc84\" (UID: \"969e9a2a-59b9-48fd-978a-9027833aeb8d\") " pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" Oct 30 18:34:34.441950 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.441924 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/969e9a2a-59b9-48fd-978a-9027833aeb8d-serving-certs-ca-bundle\") pod \"telemeter-client-5795d67875-qlc84\" (UID: \"969e9a2a-59b9-48fd-978a-9027833aeb8d\") " pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" Oct 30 18:34:34.443939 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.443846 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/969e9a2a-59b9-48fd-978a-9027833aeb8d-federate-client-tls\") pod \"telemeter-client-5795d67875-qlc84\" (UID: \"969e9a2a-59b9-48fd-978a-9027833aeb8d\") " pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" Oct 30 18:34:34.443939 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.443898 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/969e9a2a-59b9-48fd-978a-9027833aeb8d-telemeter-client-tls\") pod \"telemeter-client-5795d67875-qlc84\" (UID: \"969e9a2a-59b9-48fd-978a-9027833aeb8d\") " pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" Oct 30 18:34:34.443939 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.443898 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/969e9a2a-59b9-48fd-978a-9027833aeb8d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5795d67875-qlc84\" (UID: \"969e9a2a-59b9-48fd-978a-9027833aeb8d\") " pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" Oct 30 18:34:34.444537 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.444509 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/969e9a2a-59b9-48fd-978a-9027833aeb8d-secret-telemeter-client\") pod \"telemeter-client-5795d67875-qlc84\" (UID: \"969e9a2a-59b9-48fd-978a-9027833aeb8d\") " pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" Oct 30 18:34:34.499189 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.497260 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-chsds" podStartSLOduration=2.011621804 podStartE2EDuration="18.497238399s" totalImagesPullingTime="16.485616595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:34.495177466 +0000 UTC m=+34.522282426" watchObservedRunningTime="2026-10-30 18:34:34.497238399 +0000 UTC m=+34.524343360" Oct 30 18:34:34.504875 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.504839 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzfln\" (UniqueName: \"kubernetes.io/projected/969e9a2a-59b9-48fd-978a-9027833aeb8d-kube-api-access-wzfln\") pod \"telemeter-client-5795d67875-qlc84\" (UID: \"969e9a2a-59b9-48fd-978a-9027833aeb8d\") " pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" Oct 30 18:34:34.541034 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.540955 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" Oct 30 18:34:34.976922 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:34.976836 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5795d67875-qlc84"] Oct 30 18:34:34.981719 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:34.981681 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod969e9a2a_59b9_48fd_978a_9027833aeb8d.slice/crio-fbab68ad5dc67833adffd46c9630842c481ff23e41b0abd0f337ef6598dc2963 WatchSource:0}: Error finding container fbab68ad5dc67833adffd46c9630842c481ff23e41b0abd0f337ef6598dc2963: Status 404 returned error can't find the container with id fbab68ad5dc67833adffd46c9630842c481ff23e41b0abd0f337ef6598dc2963 Oct 30 18:34:35.284893 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:35.284856 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" event={"ID":"969e9a2a-59b9-48fd-978a-9027833aeb8d","Type":"ContainerStarted","Data":"fbab68ad5dc67833adffd46c9630842c481ff23e41b0abd0f337ef6598dc2963"} Oct 30 18:34:36.046402 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:36.046262 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hh4r5" Oct 30 18:34:37.595109 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.595074 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:34:37.620193 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.620157 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.673361 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.673327 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 18:34:37.673361 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.673336 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 18:34:37.673614 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.673471 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 18:34:37.723061 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.722999 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 18:34:37.723294 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.723000 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 18:34:37.723294 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.723278 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ccbd2\"" Oct 30 18:34:37.770150 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.770096 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-config\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.770324 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.770159 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.770324 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.770252 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.770398 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.770334 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.770398 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.770357 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.770398 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.770374 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.770516 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.770397 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbkp4\" (UniqueName: \"kubernetes.io/projected/c0ca5399-9f17-4617-8594-555e6ca9ef1f-kube-api-access-wbkp4\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.770516 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.770440 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0ca5399-9f17-4617-8594-555e6ca9ef1f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.770516 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.770485 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.770655 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.770637 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.770713 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.770672 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.770753 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.770710 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0ca5399-9f17-4617-8594-555e6ca9ef1f-config-out\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.770819 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.770796 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.770862 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.770838 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.770916 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.770865 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.770976 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.770940 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.771029 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.770976 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.771029 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.771005 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.771099 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.771037 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.771152 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.771092 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-web-config\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.773659 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.773635 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 18:34:37.774839 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.774814 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:34:37.822620 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.822583 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 18:34:37.822916 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.822677 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-67nq76prd9mfc\"" Oct 30 18:34:37.822916 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.822688 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 18:34:37.822916 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.822715 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 18:34:37.822916 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.822737 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 18:34:37.822916 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.822785 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 18:34:37.823181 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.823101 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 18:34:37.823359 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.823187 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 18:34:37.823561 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.823537 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 18:34:37.823800 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.823739 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 18:34:37.871663 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.871583 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.871663 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.871634 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.871906 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.871668 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.871906 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.871732 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.871906 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.871759 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.871906 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.871788 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.871906 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.871813 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-web-config\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.871906 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.871844 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-config\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.871906 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.871865 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.871906 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.871894 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.873711 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.871932 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.873711 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.871955 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.873711 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.871979 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.873711 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.872000 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbkp4\" (UniqueName: \"kubernetes.io/projected/c0ca5399-9f17-4617-8594-555e6ca9ef1f-kube-api-access-wbkp4\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.873711 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.872026 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0ca5399-9f17-4617-8594-555e6ca9ef1f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.873711 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.872051 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.873711 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.872092 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.873711 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.872116 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.873711 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.872169 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0ca5399-9f17-4617-8594-555e6ca9ef1f-config-out\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.873711 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.872209 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.873711 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.872736 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.873711 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.872799 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.873711 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.873103 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.873711 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.873504 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.873711 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.873551 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.874497 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.874184 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.874880 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.874852 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.879016 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.878789 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-web-config\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.879016 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.878919 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.879387 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.879315 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0ca5399-9f17-4617-8594-555e6ca9ef1f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.879609 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.879482 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0ca5399-9f17-4617-8594-555e6ca9ef1f-config-out\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.880980 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.879963 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.880980 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.879961 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.880980 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.880665 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.890558 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.882095 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.890558 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.882382 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.890558 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.882914 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.890558 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.883308 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-config\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.894781 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.894743 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:37.934371 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:37.934335 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbkp4\" (UniqueName: \"kubernetes.io/projected/c0ca5399-9f17-4617-8594-555e6ca9ef1f-kube-api-access-wbkp4\") pod \"prometheus-k8s-0\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:38.086657 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:38.086614 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:38.302329 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:38.302179 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" event={"ID":"969e9a2a-59b9-48fd-978a-9027833aeb8d","Type":"ContainerStarted","Data":"7cd9e64bd167af09fd604d8298da96d3e4218cee318495aa4f5e6100bdc49cc8"} Oct 30 18:34:38.623280 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:38.623245 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:34:38.629460 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:34:38.629430 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ca5399_9f17_4617_8594_555e6ca9ef1f.slice/crio-38aac715e4950d6640901abeb71d1665d6f2f95d45182f24a33f868dcb819c3e WatchSource:0}: Error finding container 38aac715e4950d6640901abeb71d1665d6f2f95d45182f24a33f868dcb819c3e: Status 404 returned error can't find the container with id 38aac715e4950d6640901abeb71d1665d6f2f95d45182f24a33f868dcb819c3e Oct 30 18:34:39.219805 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:39.219772 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b4bd86cfb-cfpnq" Oct 30 18:34:39.308988 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:39.308946 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0ca5399-9f17-4617-8594-555e6ca9ef1f","Type":"ContainerStarted","Data":"38aac715e4950d6640901abeb71d1665d6f2f95d45182f24a33f868dcb819c3e"} Oct 30 18:34:41.318080 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:41.318039 4531 generic.go:356] "Generic (PLEG): container finished" podID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerID="7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2" exitCode=0 Oct 30 18:34:41.318606 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:41.318147 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0ca5399-9f17-4617-8594-555e6ca9ef1f","Type":"ContainerDied","Data":"7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2"} Oct 30 18:34:41.320869 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:41.320843 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" event={"ID":"969e9a2a-59b9-48fd-978a-9027833aeb8d","Type":"ContainerStarted","Data":"d46f9d07738a2115d5a7756925f69a4de1020a5ba54e0cb1fa9b5ef86a8f0ea3"} Oct 30 18:34:41.320965 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:41.320901 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" event={"ID":"969e9a2a-59b9-48fd-978a-9027833aeb8d","Type":"ContainerStarted","Data":"4aea980bf8f8da18b0f894af8f86f217792c9aec9809f64abe70c10ca77b302a"} Oct 30 18:34:41.570872 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:41.570759 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5795d67875-qlc84" podStartSLOduration=2.104249766 podStartE2EDuration="7.570741471s" totalImagesPullingTime="5.466491705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:41.569084479 +0000 UTC m=+41.596189453" watchObservedRunningTime="2026-10-30 18:34:41.570741471 +0000 UTC m=+41.597846429" Oct 30 18:34:46.341444 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:46.341412 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0ca5399-9f17-4617-8594-555e6ca9ef1f","Type":"ContainerStarted","Data":"c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15"} Oct 30 18:34:46.341861 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:46.341477 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0ca5399-9f17-4617-8594-555e6ca9ef1f","Type":"ContainerStarted","Data":"940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4"} Oct 30 18:34:49.356031 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:49.355996 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0ca5399-9f17-4617-8594-555e6ca9ef1f","Type":"ContainerStarted","Data":"d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec"} Oct 30 18:34:49.356504 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:49.356077 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0ca5399-9f17-4617-8594-555e6ca9ef1f","Type":"ContainerStarted","Data":"1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e"} Oct 30 18:34:49.356504 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:49.356156 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0ca5399-9f17-4617-8594-555e6ca9ef1f","Type":"ContainerStarted","Data":"8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a"} Oct 30 18:34:49.356504 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:49.356198 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0ca5399-9f17-4617-8594-555e6ca9ef1f","Type":"ContainerStarted","Data":"e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262"} Oct 30 18:34:49.638981 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:49.638926 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.304894333 podStartE2EDuration="12.638907708s" totalImagesPullingTime="9.293655661s" totalInitContainerRuntime="40.357714ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:49.588535796 +0000 UTC m=+49.615640754" watchObservedRunningTime="2026-10-30 18:34:49.638907708 +0000 UTC m=+49.666012667" Oct 30 18:34:53.087560 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:34:53.087514 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:56.398853 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:34:56.398815 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:35:07.361705 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:07.361662 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-zhmhr-2v9dv"] Oct 30 18:35:07.362159 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:07.361921 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" podUID="10f6da88-0f08-47df-aa8f-7594b0dd3cc3" containerName="insights-gathering" containerID="cri-o://dd0e0940c9df298df8aa64d46bcfaf3df7c7a33ebd904bd673f277a3179d4cc7" gracePeriod=30 Oct 30 18:35:07.676892 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:07.676853 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" Oct 30 18:35:07.719115 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:07.719085 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-archives-path\") pod \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\" (UID: \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\") " Oct 30 18:35:07.719303 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:07.719170 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-trusted-ca-bundle\") pod \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\" (UID: \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\") " Oct 30 18:35:07.719303 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:07.719229 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-service-ca-bundle\") pod \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\" (UID: \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\") " Oct 30 18:35:07.719303 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:07.719264 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-kube-api-access-5rswb\" (UniqueName: \"kubernetes.io/projected/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-kube-api-access-5rswb\") pod \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\" (UID: \"10f6da88-0f08-47df-aa8f-7594b0dd3cc3\") " Oct 30 18:35:07.719526 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:07.719501 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-archives-path" pod "10f6da88-0f08-47df-aa8f-7594b0dd3cc3" (UID: "10f6da88-0f08-47df-aa8f-7594b0dd3cc3"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:35:07.719599 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:07.719534 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-trusted-ca-bundle" pod "10f6da88-0f08-47df-aa8f-7594b0dd3cc3" (UID: "10f6da88-0f08-47df-aa8f-7594b0dd3cc3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:35:07.719599 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:07.719538 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-service-ca-bundle" pod "10f6da88-0f08-47df-aa8f-7594b0dd3cc3" (UID: "10f6da88-0f08-47df-aa8f-7594b0dd3cc3"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:35:07.721817 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:07.721787 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-kube-api-access-5rswb" pod "10f6da88-0f08-47df-aa8f-7594b0dd3cc3" (UID: "10f6da88-0f08-47df-aa8f-7594b0dd3cc3"). InnerVolumeSpecName "kube-api-access-5rswb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:35:07.820312 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:07.820272 4531 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-service-ca-bundle\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:35:07.820312 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:07.820303 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rswb\" (UniqueName: \"kubernetes.io/projected/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-kube-api-access-5rswb\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:35:07.820312 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:07.820316 4531 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-archives-path\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:35:07.820312 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:07.820324 4531 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10f6da88-0f08-47df-aa8f-7594b0dd3cc3-trusted-ca-bundle\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:35:08.429315 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:08.429224 4531 generic.go:356] "Generic (PLEG): container finished" podID="10f6da88-0f08-47df-aa8f-7594b0dd3cc3" containerID="dd0e0940c9df298df8aa64d46bcfaf3df7c7a33ebd904bd673f277a3179d4cc7" exitCode=2 Oct 30 18:35:08.429315 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:08.429282 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" Oct 30 18:35:08.429791 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:08.429317 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" event={"ID":"10f6da88-0f08-47df-aa8f-7594b0dd3cc3","Type":"ContainerDied","Data":"dd0e0940c9df298df8aa64d46bcfaf3df7c7a33ebd904bd673f277a3179d4cc7"} Oct 30 18:35:08.429791 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:08.429379 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-zhmhr-2v9dv" event={"ID":"10f6da88-0f08-47df-aa8f-7594b0dd3cc3","Type":"ContainerDied","Data":"44d18475766b206efd944f4e66107fafe7cc348b214cd44c7a889f8a58b44789"} Oct 30 18:35:08.429791 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:08.429401 4531 scope.go:122] "RemoveContainer" containerID="dd0e0940c9df298df8aa64d46bcfaf3df7c7a33ebd904bd673f277a3179d4cc7" Oct 30 18:35:08.449395 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:08.449371 4531 scope.go:122] "RemoveContainer" containerID="dd0e0940c9df298df8aa64d46bcfaf3df7c7a33ebd904bd673f277a3179d4cc7" Oct 30 18:35:08.449757 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:35:08.449726 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd0e0940c9df298df8aa64d46bcfaf3df7c7a33ebd904bd673f277a3179d4cc7\": container with ID starting with dd0e0940c9df298df8aa64d46bcfaf3df7c7a33ebd904bd673f277a3179d4cc7 not found: ID does not exist" containerID="dd0e0940c9df298df8aa64d46bcfaf3df7c7a33ebd904bd673f277a3179d4cc7" Oct 30 18:35:08.449865 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:08.449767 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd0e0940c9df298df8aa64d46bcfaf3df7c7a33ebd904bd673f277a3179d4cc7"} err="failed to get container status \"dd0e0940c9df298df8aa64d46bcfaf3df7c7a33ebd904bd673f277a3179d4cc7\": rpc error: code = NotFound desc = could not find container \"dd0e0940c9df298df8aa64d46bcfaf3df7c7a33ebd904bd673f277a3179d4cc7\": container with ID starting with dd0e0940c9df298df8aa64d46bcfaf3df7c7a33ebd904bd673f277a3179d4cc7 not found: ID does not exist" Oct 30 18:35:08.595175 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:08.594058 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-zhmhr-2v9dv"] Oct 30 18:35:08.642605 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:08.642565 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-zhmhr-2v9dv"] Oct 30 18:35:10.099511 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:10.099475 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10f6da88-0f08-47df-aa8f-7594b0dd3cc3" path="/var/lib/kubelet/pods/10f6da88-0f08-47df-aa8f-7594b0dd3cc3/volumes" Oct 30 18:35:26.398801 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:35:26.398761 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:35:45.373096 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:45.373047 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:35:45.393192 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:45.393162 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:35:45.576384 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:35:45.576355 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:35:56.399548 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:35:56.399508 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:36:00.654163 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.654107 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:36:00.654728 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.654552 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerName="prometheus" containerID="cri-o://940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4" gracePeriod=600 Oct 30 18:36:00.654728 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.654588 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerName="thanos-sidecar" containerID="cri-o://e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262" gracePeriod=600 Oct 30 18:36:00.654728 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.654630 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec" gracePeriod=600 Oct 30 18:36:00.654728 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.654684 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerName="kube-rbac-proxy-web" containerID="cri-o://8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a" gracePeriod=600 Oct 30 18:36:00.654728 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.654557 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerName="kube-rbac-proxy" containerID="cri-o://1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e" gracePeriod=600 Oct 30 18:36:00.655352 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.655290 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerName="config-reloader" containerID="cri-o://c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15" gracePeriod=600 Oct 30 18:36:00.911464 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.911390 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:00.951360 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.951324 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-configmap-kubelet-serving-ca-bundle\") pod \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " Oct 30 18:36:00.951470 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.951386 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-configmap-metrics-client-ca\") pod \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " Oct 30 18:36:00.951470 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.951411 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-grpc-tls\") pod \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " Oct 30 18:36:00.951470 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.951432 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-rulefiles-1\") pod \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " Oct 30 18:36:00.951470 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.951465 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ca5399-9f17-4617-8594-555e6ca9ef1f-kube-api-access-wbkp4\" (UniqueName: \"kubernetes.io/projected/c0ca5399-9f17-4617-8594-555e6ca9ef1f-kube-api-access-wbkp4\") pod \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " Oct 30 18:36:00.951665 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.951482 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-rulefiles-0\") pod \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " Oct 30 18:36:00.951665 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.951500 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-web-config\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-web-config\") pod \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " Oct 30 18:36:00.951665 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.951522 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-db\") pod \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " Oct 30 18:36:00.951665 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.951546 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-prometheus-k8s-tls\") pod \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " Oct 30 18:36:00.951665 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.951576 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-kube-rbac-proxy\") pod \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " Oct 30 18:36:00.951665 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.951613 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " Oct 30 18:36:00.951665 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.951657 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ca5399-9f17-4617-8594-555e6ca9ef1f-tls-assets\" (UniqueName: \"kubernetes.io/projected/c0ca5399-9f17-4617-8594-555e6ca9ef1f-tls-assets\") pod \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " Oct 30 18:36:00.952044 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.951714 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-rulefiles-2\") pod \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " Oct 30 18:36:00.952044 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.951746 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-trusted-ca-bundle\") pod \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " Oct 30 18:36:00.952044 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.951773 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " Oct 30 18:36:00.952044 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.951811 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-config\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-config\") pod \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " Oct 30 18:36:00.952044 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.951822 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-configmap-metrics-client-ca" pod "c0ca5399-9f17-4617-8594-555e6ca9ef1f" (UID: "c0ca5399-9f17-4617-8594-555e6ca9ef1f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:00.952044 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.951839 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-thanos-prometheus-http-client-file\") pod \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " Oct 30 18:36:00.952044 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.951921 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-metrics-client-certs\") pod \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " Oct 30 18:36:00.952044 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.951960 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ca5399-9f17-4617-8594-555e6ca9ef1f-config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0ca5399-9f17-4617-8594-555e6ca9ef1f-config-out\") pod \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " Oct 30 18:36:00.952482 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.952084 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-configmap-kubelet-serving-ca-bundle" pod "c0ca5399-9f17-4617-8594-555e6ca9ef1f" (UID: "c0ca5399-9f17-4617-8594-555e6ca9ef1f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:00.952482 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.952155 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-configmap-serving-certs-ca-bundle\") pod \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\" (UID: \"c0ca5399-9f17-4617-8594-555e6ca9ef1f\") " Oct 30 18:36:00.952482 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.952182 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-rulefiles-1" pod "c0ca5399-9f17-4617-8594-555e6ca9ef1f" (UID: "c0ca5399-9f17-4617-8594-555e6ca9ef1f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:00.952482 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.952414 4531 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:36:00.952482 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.952442 4531 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-configmap-metrics-client-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:36:00.952482 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.952457 4531 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:36:00.952971 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.952943 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-configmap-serving-certs-ca-bundle" pod "c0ca5399-9f17-4617-8594-555e6ca9ef1f" (UID: "c0ca5399-9f17-4617-8594-555e6ca9ef1f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:00.953384 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.953360 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-rulefiles-2" pod "c0ca5399-9f17-4617-8594-555e6ca9ef1f" (UID: "c0ca5399-9f17-4617-8594-555e6ca9ef1f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:00.954184 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.953746 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-rulefiles-0" pod "c0ca5399-9f17-4617-8594-555e6ca9ef1f" (UID: "c0ca5399-9f17-4617-8594-555e6ca9ef1f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:00.955017 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.954877 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-trusted-ca-bundle" pod "c0ca5399-9f17-4617-8594-555e6ca9ef1f" (UID: "c0ca5399-9f17-4617-8594-555e6ca9ef1f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:00.955285 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.955254 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ca5399-9f17-4617-8594-555e6ca9ef1f-kube-api-access-wbkp4" pod "c0ca5399-9f17-4617-8594-555e6ca9ef1f" (UID: "c0ca5399-9f17-4617-8594-555e6ca9ef1f"). InnerVolumeSpecName "kube-api-access-wbkp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:00.955606 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.955571 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-prometheus-k8s-tls" pod "c0ca5399-9f17-4617-8594-555e6ca9ef1f" (UID: "c0ca5399-9f17-4617-8594-555e6ca9ef1f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:00.955733 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.955694 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-thanos-prometheus-http-client-file" pod "c0ca5399-9f17-4617-8594-555e6ca9ef1f" (UID: "c0ca5399-9f17-4617-8594-555e6ca9ef1f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:00.955733 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.955712 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-grpc-tls" pod "c0ca5399-9f17-4617-8594-555e6ca9ef1f" (UID: "c0ca5399-9f17-4617-8594-555e6ca9ef1f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:00.956048 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.955997 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ca5399-9f17-4617-8594-555e6ca9ef1f-config-out" pod "c0ca5399-9f17-4617-8594-555e6ca9ef1f" (UID: "c0ca5399-9f17-4617-8594-555e6ca9ef1f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:00.956166 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.956055 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-db" pod "c0ca5399-9f17-4617-8594-555e6ca9ef1f" (UID: "c0ca5399-9f17-4617-8594-555e6ca9ef1f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:00.957290 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.957259 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-metrics-client-certs" pod "c0ca5399-9f17-4617-8594-555e6ca9ef1f" (UID: "c0ca5399-9f17-4617-8594-555e6ca9ef1f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:00.957381 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.957333 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-prometheus-k8s-kube-rbac-proxy-web" pod "c0ca5399-9f17-4617-8594-555e6ca9ef1f" (UID: "c0ca5399-9f17-4617-8594-555e6ca9ef1f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:00.957669 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.957648 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-kube-rbac-proxy" pod "c0ca5399-9f17-4617-8594-555e6ca9ef1f" (UID: "c0ca5399-9f17-4617-8594-555e6ca9ef1f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:00.957746 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.957675 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ca5399-9f17-4617-8594-555e6ca9ef1f-tls-assets" pod "c0ca5399-9f17-4617-8594-555e6ca9ef1f" (UID: "c0ca5399-9f17-4617-8594-555e6ca9ef1f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:00.957794 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.957747 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-config" pod "c0ca5399-9f17-4617-8594-555e6ca9ef1f" (UID: "c0ca5399-9f17-4617-8594-555e6ca9ef1f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:00.957794 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.957768 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-prometheus-k8s-thanos-sidecar-tls" pod "c0ca5399-9f17-4617-8594-555e6ca9ef1f" (UID: "c0ca5399-9f17-4617-8594-555e6ca9ef1f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:00.968848 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:00.968806 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-web-config" pod "c0ca5399-9f17-4617-8594-555e6ca9ef1f" (UID: "c0ca5399-9f17-4617-8594-555e6ca9ef1f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:01.053225 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.053184 4531 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:36:01.053225 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.053228 4531 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0ca5399-9f17-4617-8594-555e6ca9ef1f-tls-assets\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:36:01.053488 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.053244 4531 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:36:01.053488 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.053260 4531 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:36:01.053488 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.053276 4531 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:36:01.053488 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.053291 4531 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-config\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:36:01.053488 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.053306 4531 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:36:01.053488 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.053321 4531 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-metrics-client-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:36:01.053488 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.053334 4531 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0ca5399-9f17-4617-8594-555e6ca9ef1f-config-out\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:36:01.053488 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.053347 4531 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:36:01.053488 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.053360 4531 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-grpc-tls\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:36:01.053488 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.053372 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbkp4\" (UniqueName: \"kubernetes.io/projected/c0ca5399-9f17-4617-8594-555e6ca9ef1f-kube-api-access-wbkp4\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:36:01.053488 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.053385 4531 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:36:01.053488 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.053397 4531 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-web-config\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:36:01.053488 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.053406 4531 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0ca5399-9f17-4617-8594-555e6ca9ef1f-prometheus-k8s-db\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:36:01.053488 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.053414 4531 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:36:01.053488 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.053423 4531 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0ca5399-9f17-4617-8594-555e6ca9ef1f-secret-kube-rbac-proxy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:36:01.618461 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.618423 4531 generic.go:356] "Generic (PLEG): container finished" podID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerID="d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec" exitCode=0 Oct 30 18:36:01.618461 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.618452 4531 generic.go:356] "Generic (PLEG): container finished" podID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerID="1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e" exitCode=0 Oct 30 18:36:01.618461 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.618458 4531 generic.go:356] "Generic (PLEG): container finished" podID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerID="8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a" exitCode=0 Oct 30 18:36:01.618461 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.618465 4531 generic.go:356] "Generic (PLEG): container finished" podID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerID="e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262" exitCode=0 Oct 30 18:36:01.618461 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.618470 4531 generic.go:356] "Generic (PLEG): container finished" podID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerID="c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15" exitCode=0 Oct 30 18:36:01.618461 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.618475 4531 generic.go:356] "Generic (PLEG): container finished" podID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerID="940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4" exitCode=0 Oct 30 18:36:01.618929 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.618576 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0ca5399-9f17-4617-8594-555e6ca9ef1f","Type":"ContainerDied","Data":"d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec"} Oct 30 18:36:01.618929 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.618598 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:01.618929 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.618668 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0ca5399-9f17-4617-8594-555e6ca9ef1f","Type":"ContainerDied","Data":"1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e"} Oct 30 18:36:01.618929 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.618719 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0ca5399-9f17-4617-8594-555e6ca9ef1f","Type":"ContainerDied","Data":"8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a"} Oct 30 18:36:01.618929 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.618738 4531 scope.go:122] "RemoveContainer" containerID="d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec" Oct 30 18:36:01.618929 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.618761 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0ca5399-9f17-4617-8594-555e6ca9ef1f","Type":"ContainerDied","Data":"e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262"} Oct 30 18:36:01.618929 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.618805 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0ca5399-9f17-4617-8594-555e6ca9ef1f","Type":"ContainerDied","Data":"c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15"} Oct 30 18:36:01.618929 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.618849 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0ca5399-9f17-4617-8594-555e6ca9ef1f","Type":"ContainerDied","Data":"940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4"} Oct 30 18:36:01.618929 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.618891 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0ca5399-9f17-4617-8594-555e6ca9ef1f","Type":"ContainerDied","Data":"38aac715e4950d6640901abeb71d1665d6f2f95d45182f24a33f868dcb819c3e"} Oct 30 18:36:01.627029 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.627007 4531 scope.go:122] "RemoveContainer" containerID="1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e" Oct 30 18:36:01.635059 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.635030 4531 scope.go:122] "RemoveContainer" containerID="8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a" Oct 30 18:36:01.643177 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.643152 4531 scope.go:122] "RemoveContainer" containerID="e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262" Oct 30 18:36:01.651239 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.651217 4531 scope.go:122] "RemoveContainer" containerID="c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15" Oct 30 18:36:01.659243 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.659223 4531 scope.go:122] "RemoveContainer" containerID="940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4" Oct 30 18:36:01.681244 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.681217 4531 scope.go:122] "RemoveContainer" containerID="7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2" Oct 30 18:36:01.689402 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.689377 4531 scope.go:122] "RemoveContainer" containerID="d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec" Oct 30 18:36:01.689758 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:36:01.689737 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec\": container with ID starting with d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec not found: ID does not exist" containerID="d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec" Oct 30 18:36:01.689804 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.689770 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec"} err="failed to get container status \"d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec\": rpc error: code = NotFound desc = could not find container \"d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec\": container with ID starting with d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec not found: ID does not exist" Oct 30 18:36:01.689804 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.689792 4531 scope.go:122] "RemoveContainer" containerID="1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e" Oct 30 18:36:01.690054 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:36:01.690036 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e\": container with ID starting with 1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e not found: ID does not exist" containerID="1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e" Oct 30 18:36:01.690103 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.690058 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e"} err="failed to get container status \"1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e\": rpc error: code = NotFound desc = could not find container \"1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e\": container with ID starting with 1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e not found: ID does not exist" Oct 30 18:36:01.690103 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.690072 4531 scope.go:122] "RemoveContainer" containerID="8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a" Oct 30 18:36:01.690682 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:36:01.690664 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a\": container with ID starting with 8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a not found: ID does not exist" containerID="8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a" Oct 30 18:36:01.690755 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.690694 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a"} err="failed to get container status \"8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a\": rpc error: code = NotFound desc = could not find container \"8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a\": container with ID starting with 8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a not found: ID does not exist" Oct 30 18:36:01.690755 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.690718 4531 scope.go:122] "RemoveContainer" containerID="e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262" Oct 30 18:36:01.690992 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:36:01.690975 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262\": container with ID starting with e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262 not found: ID does not exist" containerID="e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262" Oct 30 18:36:01.691027 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.691000 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262"} err="failed to get container status \"e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262\": rpc error: code = NotFound desc = could not find container \"e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262\": container with ID starting with e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262 not found: ID does not exist" Oct 30 18:36:01.691027 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.691013 4531 scope.go:122] "RemoveContainer" containerID="c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15" Oct 30 18:36:01.691254 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:36:01.691233 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15\": container with ID starting with c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15 not found: ID does not exist" containerID="c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15" Oct 30 18:36:01.691328 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.691261 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15"} err="failed to get container status \"c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15\": rpc error: code = NotFound desc = could not find container \"c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15\": container with ID starting with c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15 not found: ID does not exist" Oct 30 18:36:01.691328 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.691280 4531 scope.go:122] "RemoveContainer" containerID="940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4" Oct 30 18:36:01.691500 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:36:01.691484 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4\": container with ID starting with 940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4 not found: ID does not exist" containerID="940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4" Oct 30 18:36:01.691540 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.691504 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4"} err="failed to get container status \"940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4\": rpc error: code = NotFound desc = could not find container \"940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4\": container with ID starting with 940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4 not found: ID does not exist" Oct 30 18:36:01.691540 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.691516 4531 scope.go:122] "RemoveContainer" containerID="7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2" Oct 30 18:36:01.691732 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:36:01.691713 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2\": container with ID starting with 7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2 not found: ID does not exist" containerID="7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2" Oct 30 18:36:01.691800 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.691739 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2"} err="failed to get container status \"7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2\": rpc error: code = NotFound desc = could not find container \"7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2\": container with ID starting with 7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2 not found: ID does not exist" Oct 30 18:36:01.691800 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.691760 4531 scope.go:122] "RemoveContainer" containerID="d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec" Oct 30 18:36:01.691998 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.691982 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec"} err="failed to get container status \"d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec\": rpc error: code = NotFound desc = could not find container \"d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec\": container with ID starting with d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec not found: ID does not exist" Oct 30 18:36:01.692045 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.691998 4531 scope.go:122] "RemoveContainer" containerID="1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e" Oct 30 18:36:01.692234 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.692212 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e"} err="failed to get container status \"1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e\": rpc error: code = NotFound desc = could not find container \"1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e\": container with ID starting with 1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e not found: ID does not exist" Oct 30 18:36:01.692312 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.692235 4531 scope.go:122] "RemoveContainer" containerID="8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a" Oct 30 18:36:01.692508 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.692490 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a"} err="failed to get container status \"8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a\": rpc error: code = NotFound desc = could not find container \"8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a\": container with ID starting with 8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a not found: ID does not exist" Oct 30 18:36:01.692552 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.692508 4531 scope.go:122] "RemoveContainer" containerID="e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262" Oct 30 18:36:01.692724 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.692705 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262"} err="failed to get container status \"e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262\": rpc error: code = NotFound desc = could not find container \"e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262\": container with ID starting with e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262 not found: ID does not exist" Oct 30 18:36:01.692794 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.692725 4531 scope.go:122] "RemoveContainer" containerID="c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15" Oct 30 18:36:01.692956 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.692940 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15"} err="failed to get container status \"c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15\": rpc error: code = NotFound desc = could not find container \"c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15\": container with ID starting with c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15 not found: ID does not exist" Oct 30 18:36:01.693002 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.692957 4531 scope.go:122] "RemoveContainer" containerID="940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4" Oct 30 18:36:01.693183 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.693163 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4"} err="failed to get container status \"940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4\": rpc error: code = NotFound desc = could not find container \"940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4\": container with ID starting with 940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4 not found: ID does not exist" Oct 30 18:36:01.693250 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.693184 4531 scope.go:122] "RemoveContainer" containerID="7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2" Oct 30 18:36:01.693414 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.693397 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2"} err="failed to get container status \"7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2\": rpc error: code = NotFound desc = could not find container \"7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2\": container with ID starting with 7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2 not found: ID does not exist" Oct 30 18:36:01.693457 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.693414 4531 scope.go:122] "RemoveContainer" containerID="d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec" Oct 30 18:36:01.693613 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.693596 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec"} err="failed to get container status \"d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec\": rpc error: code = NotFound desc = could not find container \"d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec\": container with ID starting with d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec not found: ID does not exist" Oct 30 18:36:01.693697 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.693615 4531 scope.go:122] "RemoveContainer" containerID="1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e" Oct 30 18:36:01.693843 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.693826 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e"} err="failed to get container status \"1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e\": rpc error: code = NotFound desc = could not find container \"1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e\": container with ID starting with 1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e not found: ID does not exist" Oct 30 18:36:01.693885 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.693842 4531 scope.go:122] "RemoveContainer" containerID="8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a" Oct 30 18:36:01.694051 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.694034 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a"} err="failed to get container status \"8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a\": rpc error: code = NotFound desc = could not find container \"8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a\": container with ID starting with 8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a not found: ID does not exist" Oct 30 18:36:01.694120 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.694052 4531 scope.go:122] "RemoveContainer" containerID="e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262" Oct 30 18:36:01.694292 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.694277 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262"} err="failed to get container status \"e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262\": rpc error: code = NotFound desc = could not find container \"e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262\": container with ID starting with e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262 not found: ID does not exist" Oct 30 18:36:01.694292 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.694292 4531 scope.go:122] "RemoveContainer" containerID="c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15" Oct 30 18:36:01.694501 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.694483 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15"} err="failed to get container status \"c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15\": rpc error: code = NotFound desc = could not find container \"c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15\": container with ID starting with c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15 not found: ID does not exist" Oct 30 18:36:01.694562 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.694502 4531 scope.go:122] "RemoveContainer" containerID="940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4" Oct 30 18:36:01.694714 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.694698 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4"} err="failed to get container status \"940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4\": rpc error: code = NotFound desc = could not find container \"940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4\": container with ID starting with 940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4 not found: ID does not exist" Oct 30 18:36:01.694759 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.694714 4531 scope.go:122] "RemoveContainer" containerID="7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2" Oct 30 18:36:01.694919 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.694901 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2"} err="failed to get container status \"7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2\": rpc error: code = NotFound desc = could not find container \"7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2\": container with ID starting with 7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2 not found: ID does not exist" Oct 30 18:36:01.694985 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.694919 4531 scope.go:122] "RemoveContainer" containerID="d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec" Oct 30 18:36:01.695215 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.695171 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec"} err="failed to get container status \"d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec\": rpc error: code = NotFound desc = could not find container \"d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec\": container with ID starting with d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec not found: ID does not exist" Oct 30 18:36:01.695215 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.695188 4531 scope.go:122] "RemoveContainer" containerID="1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e" Oct 30 18:36:01.695369 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.695353 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e"} err="failed to get container status \"1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e\": rpc error: code = NotFound desc = could not find container \"1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e\": container with ID starting with 1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e not found: ID does not exist" Oct 30 18:36:01.695410 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.695369 4531 scope.go:122] "RemoveContainer" containerID="8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a" Oct 30 18:36:01.695562 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.695546 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a"} err="failed to get container status \"8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a\": rpc error: code = NotFound desc = could not find container \"8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a\": container with ID starting with 8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a not found: ID does not exist" Oct 30 18:36:01.695562 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.695561 4531 scope.go:122] "RemoveContainer" containerID="e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262" Oct 30 18:36:01.695764 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.695746 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262"} err="failed to get container status \"e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262\": rpc error: code = NotFound desc = could not find container \"e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262\": container with ID starting with e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262 not found: ID does not exist" Oct 30 18:36:01.695805 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.695764 4531 scope.go:122] "RemoveContainer" containerID="c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15" Oct 30 18:36:01.696027 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.696000 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15"} err="failed to get container status \"c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15\": rpc error: code = NotFound desc = could not find container \"c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15\": container with ID starting with c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15 not found: ID does not exist" Oct 30 18:36:01.696027 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.696023 4531 scope.go:122] "RemoveContainer" containerID="940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4" Oct 30 18:36:01.696304 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.696288 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4"} err="failed to get container status \"940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4\": rpc error: code = NotFound desc = could not find container \"940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4\": container with ID starting with 940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4 not found: ID does not exist" Oct 30 18:36:01.696304 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.696303 4531 scope.go:122] "RemoveContainer" containerID="7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2" Oct 30 18:36:01.696522 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.696504 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2"} err="failed to get container status \"7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2\": rpc error: code = NotFound desc = could not find container \"7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2\": container with ID starting with 7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2 not found: ID does not exist" Oct 30 18:36:01.696522 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.696521 4531 scope.go:122] "RemoveContainer" containerID="d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec" Oct 30 18:36:01.696786 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.696760 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec"} err="failed to get container status \"d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec\": rpc error: code = NotFound desc = could not find container \"d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec\": container with ID starting with d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec not found: ID does not exist" Oct 30 18:36:01.696786 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.696781 4531 scope.go:122] "RemoveContainer" containerID="1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e" Oct 30 18:36:01.697036 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.697017 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e"} err="failed to get container status \"1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e\": rpc error: code = NotFound desc = could not find container \"1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e\": container with ID starting with 1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e not found: ID does not exist" Oct 30 18:36:01.697093 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.697036 4531 scope.go:122] "RemoveContainer" containerID="8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a" Oct 30 18:36:01.697304 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.697286 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a"} err="failed to get container status \"8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a\": rpc error: code = NotFound desc = could not find container \"8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a\": container with ID starting with 8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a not found: ID does not exist" Oct 30 18:36:01.697360 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.697305 4531 scope.go:122] "RemoveContainer" containerID="e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262" Oct 30 18:36:01.697577 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.697559 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262"} err="failed to get container status \"e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262\": rpc error: code = NotFound desc = could not find container \"e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262\": container with ID starting with e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262 not found: ID does not exist" Oct 30 18:36:01.697640 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.697577 4531 scope.go:122] "RemoveContainer" containerID="c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15" Oct 30 18:36:01.697809 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.697793 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15"} err="failed to get container status \"c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15\": rpc error: code = NotFound desc = could not find container \"c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15\": container with ID starting with c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15 not found: ID does not exist" Oct 30 18:36:01.697854 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.697809 4531 scope.go:122] "RemoveContainer" containerID="940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4" Oct 30 18:36:01.698032 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.698011 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4"} err="failed to get container status \"940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4\": rpc error: code = NotFound desc = could not find container \"940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4\": container with ID starting with 940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4 not found: ID does not exist" Oct 30 18:36:01.698081 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.698032 4531 scope.go:122] "RemoveContainer" containerID="7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2" Oct 30 18:36:01.698265 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.698249 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2"} err="failed to get container status \"7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2\": rpc error: code = NotFound desc = could not find container \"7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2\": container with ID starting with 7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2 not found: ID does not exist" Oct 30 18:36:01.698265 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.698264 4531 scope.go:122] "RemoveContainer" containerID="d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec" Oct 30 18:36:01.698484 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.698466 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec"} err="failed to get container status \"d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec\": rpc error: code = NotFound desc = could not find container \"d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec\": container with ID starting with d5432da2340cf1c9e0a6195395e6afd0ab1bf0f88ba11f25d1b84c2410b082ec not found: ID does not exist" Oct 30 18:36:01.698530 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.698483 4531 scope.go:122] "RemoveContainer" containerID="1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e" Oct 30 18:36:01.698704 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.698690 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e"} err="failed to get container status \"1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e\": rpc error: code = NotFound desc = could not find container \"1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e\": container with ID starting with 1e5da4c7b916f633640a4898d71e293052a13e93983e5684e9932f81d23c0a5e not found: ID does not exist" Oct 30 18:36:01.698704 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.698704 4531 scope.go:122] "RemoveContainer" containerID="8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a" Oct 30 18:36:01.698917 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.698900 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a"} err="failed to get container status \"8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a\": rpc error: code = NotFound desc = could not find container \"8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a\": container with ID starting with 8913960dfc3919f47d404392fd727259b9a95b7c0b217e512b7b3d062c94724a not found: ID does not exist" Oct 30 18:36:01.698966 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.698917 4531 scope.go:122] "RemoveContainer" containerID="e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262" Oct 30 18:36:01.699114 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.699098 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262"} err="failed to get container status \"e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262\": rpc error: code = NotFound desc = could not find container \"e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262\": container with ID starting with e9be2a66f5e41be712df073ed9ae4ff188a4c252b626c9911670f1ef07068262 not found: ID does not exist" Oct 30 18:36:01.699178 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.699116 4531 scope.go:122] "RemoveContainer" containerID="c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15" Oct 30 18:36:01.699354 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.699337 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15"} err="failed to get container status \"c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15\": rpc error: code = NotFound desc = could not find container \"c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15\": container with ID starting with c4d58cde37c942aad40049d8256df734905ea45c324c715bd90422d3e78b5a15 not found: ID does not exist" Oct 30 18:36:01.699406 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.699354 4531 scope.go:122] "RemoveContainer" containerID="940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4" Oct 30 18:36:01.699569 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.699551 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4"} err="failed to get container status \"940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4\": rpc error: code = NotFound desc = could not find container \"940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4\": container with ID starting with 940708db6819abce155e3366a072ad89543b8eecf8631a4b04c17bced077b8d4 not found: ID does not exist" Oct 30 18:36:01.699614 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.699570 4531 scope.go:122] "RemoveContainer" containerID="7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2" Oct 30 18:36:01.699789 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.699773 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2"} err="failed to get container status \"7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2\": rpc error: code = NotFound desc = could not find container \"7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2\": container with ID starting with 7210581006a69c63d98e0f92a9cac5f41b27201a1c3ae967f293629c1a4d78c2 not found: ID does not exist" Oct 30 18:36:01.785737 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.785705 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:36:01.841702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.840379 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:36:01.841702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.840464 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:36:01.841702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.840943 4531 cpu_manager.go:414] "containerMap: removing container" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerName="init-config-reloader" Oct 30 18:36:01.841702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.840960 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerName="init-config-reloader" Oct 30 18:36:01.841702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.840991 4531 cpu_manager.go:414] "containerMap: removing container" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerName="prometheus" Oct 30 18:36:01.841702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.840998 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerName="prometheus" Oct 30 18:36:01.841702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.841009 4531 cpu_manager.go:414] "containerMap: removing container" podUID="10f6da88-0f08-47df-aa8f-7594b0dd3cc3" containerName="insights-gathering" Oct 30 18:36:01.841702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.841017 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f6da88-0f08-47df-aa8f-7594b0dd3cc3" containerName="insights-gathering" Oct 30 18:36:01.841702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.841033 4531 cpu_manager.go:414] "containerMap: removing container" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerName="config-reloader" Oct 30 18:36:01.841702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.841039 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerName="config-reloader" Oct 30 18:36:01.841702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.841049 4531 cpu_manager.go:414] "containerMap: removing container" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerName="kube-rbac-proxy" Oct 30 18:36:01.841702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.841055 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerName="kube-rbac-proxy" Oct 30 18:36:01.841702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.841088 4531 cpu_manager.go:414] "containerMap: removing container" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerName="kube-rbac-proxy-thanos" Oct 30 18:36:01.841702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.841098 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerName="kube-rbac-proxy-thanos" Oct 30 18:36:01.841702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.841147 4531 cpu_manager.go:414] "containerMap: removing container" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerName="thanos-sidecar" Oct 30 18:36:01.841702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.841154 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerName="thanos-sidecar" Oct 30 18:36:01.841702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.841164 4531 cpu_manager.go:414] "containerMap: removing container" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerName="kube-rbac-proxy-web" Oct 30 18:36:01.841702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.841170 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerName="kube-rbac-proxy-web" Oct 30 18:36:01.841702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.841282 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerName="kube-rbac-proxy-web" Oct 30 18:36:01.841702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.841294 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerName="prometheus" Oct 30 18:36:01.841702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.841301 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerName="config-reloader" Oct 30 18:36:01.841702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.841314 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f6da88-0f08-47df-aa8f-7594b0dd3cc3" containerName="insights-gathering" Oct 30 18:36:01.841702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.841324 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerName="kube-rbac-proxy" Oct 30 18:36:01.841702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.841337 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerName="kube-rbac-proxy-thanos" Oct 30 18:36:01.841702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.841347 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" containerName="thanos-sidecar" Oct 30 18:36:01.845870 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.845841 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:01.909097 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.909003 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 18:36:01.909362 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.909343 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 18:36:01.909553 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.909479 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 18:36:01.909553 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.909489 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 18:36:01.909553 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.909493 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 18:36:01.909553 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.909530 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 18:36:01.909842 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.909562 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 18:36:01.909842 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.909565 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 18:36:01.909842 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.909600 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ccbd2\"" Oct 30 18:36:01.909842 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.909659 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 18:36:01.909842 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.909741 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-67nq76prd9mfc\"" Oct 30 18:36:01.909842 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.909811 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 18:36:01.909842 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.909821 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 18:36:01.910047 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.909876 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 18:36:01.910109 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.910089 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 18:36:01.959682 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.959637 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bc2edd37-18e4-4982-8758-c45199e53612-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:01.959845 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.959692 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bc2edd37-18e4-4982-8758-c45199e53612-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:01.959845 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.959780 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:01.959845 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.959797 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:01.959845 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.959824 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-web-config\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:01.959977 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.959850 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc2edd37-18e4-4982-8758-c45199e53612-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:01.959977 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.959931 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:01.959977 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.959958 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc2edd37-18e4-4982-8758-c45199e53612-config-out\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:01.960084 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.959981 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc2edd37-18e4-4982-8758-c45199e53612-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:01.960084 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.960001 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc2edd37-18e4-4982-8758-c45199e53612-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:01.960084 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.960081 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bc2edd37-18e4-4982-8758-c45199e53612-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:01.960210 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.960096 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcm94\" (UniqueName: \"kubernetes.io/projected/bc2edd37-18e4-4982-8758-c45199e53612-kube-api-access-lcm94\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:01.960210 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.960100 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 18:36:01.960210 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.960116 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 18:36:01.960210 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.960162 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:01.960210 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.960195 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:01.960349 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.960218 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc2edd37-18e4-4982-8758-c45199e53612-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:01.960349 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.960258 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bc2edd37-18e4-4982-8758-c45199e53612-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:01.960349 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.960280 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:01.960349 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.960297 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bc2edd37-18e4-4982-8758-c45199e53612-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:01.960458 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.960352 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-config\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:01.960458 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.960378 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:01.978509 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:01.978473 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:36:02.061275 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.061226 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bc2edd37-18e4-4982-8758-c45199e53612-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.061275 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.061272 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bc2edd37-18e4-4982-8758-c45199e53612-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.061500 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.061292 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.061500 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.061399 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.061500 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.061433 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-web-config\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.061656 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.061582 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc2edd37-18e4-4982-8758-c45199e53612-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.061708 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.061649 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.061708 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.061683 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc2edd37-18e4-4982-8758-c45199e53612-config-out\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.061807 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.061714 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc2edd37-18e4-4982-8758-c45199e53612-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.061807 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.061748 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc2edd37-18e4-4982-8758-c45199e53612-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.061901 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.061803 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bc2edd37-18e4-4982-8758-c45199e53612-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.061901 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.061831 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcm94\" (UniqueName: \"kubernetes.io/projected/bc2edd37-18e4-4982-8758-c45199e53612-kube-api-access-lcm94\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.061901 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.061856 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.061901 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.061879 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.062084 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.061903 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bc2edd37-18e4-4982-8758-c45199e53612-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.062084 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.061908 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc2edd37-18e4-4982-8758-c45199e53612-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.062084 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.061969 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bc2edd37-18e4-4982-8758-c45199e53612-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.062084 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.061999 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.062084 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.062027 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bc2edd37-18e4-4982-8758-c45199e53612-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.062084 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.062070 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-config\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.062428 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.062097 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.063098 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.062918 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bc2edd37-18e4-4982-8758-c45199e53612-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.063630 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.063362 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc2edd37-18e4-4982-8758-c45199e53612-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.063630 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.063481 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bc2edd37-18e4-4982-8758-c45199e53612-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.063630 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.063572 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc2edd37-18e4-4982-8758-c45199e53612-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.064049 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.063999 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc2edd37-18e4-4982-8758-c45199e53612-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.064934 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.064681 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc2edd37-18e4-4982-8758-c45199e53612-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.064934 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.064831 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-web-config\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.065765 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.065717 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.065981 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.065953 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.066091 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.065999 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.066091 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.066026 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.067686 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.067639 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bc2edd37-18e4-4982-8758-c45199e53612-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.067901 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.067875 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-config\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.067970 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.067877 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.067970 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.067947 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.068180 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.068158 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bc2edd37-18e4-4982-8758-c45199e53612-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.068270 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.068184 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc2edd37-18e4-4982-8758-c45199e53612-config-out\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.069742 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.069723 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bc2edd37-18e4-4982-8758-c45199e53612-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.097245 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.097206 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0ca5399-9f17-4617-8594-555e6ca9ef1f" path="/var/lib/kubelet/pods/c0ca5399-9f17-4617-8594-555e6ca9ef1f/volumes" Oct 30 18:36:02.125400 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.125368 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcm94\" (UniqueName: \"kubernetes.io/projected/bc2edd37-18e4-4982-8758-c45199e53612-kube-api-access-lcm94\") pod \"prometheus-k8s-0\" (UID: \"bc2edd37-18e4-4982-8758-c45199e53612\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.162925 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.162846 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:02.596516 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.596446 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:36:02.600779 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:36:02.600753 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc2edd37_18e4_4982_8758_c45199e53612.slice/crio-8c2f2081f7fcf26a95ddc5c8ec959b4d0d97ec1679bbf43b605d40a575063087 WatchSource:0}: Error finding container 8c2f2081f7fcf26a95ddc5c8ec959b4d0d97ec1679bbf43b605d40a575063087: Status 404 returned error can't find the container with id 8c2f2081f7fcf26a95ddc5c8ec959b4d0d97ec1679bbf43b605d40a575063087 Oct 30 18:36:02.625664 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:02.625616 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc2edd37-18e4-4982-8758-c45199e53612","Type":"ContainerStarted","Data":"8c2f2081f7fcf26a95ddc5c8ec959b4d0d97ec1679bbf43b605d40a575063087"} Oct 30 18:36:03.631501 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:03.631462 4531 generic.go:356] "Generic (PLEG): container finished" podID="bc2edd37-18e4-4982-8758-c45199e53612" containerID="ff2a520a5bc05f14a3950eb886abea3e6f45b094ecb18599cc384ef4c8e4be99" exitCode=0 Oct 30 18:36:03.631926 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:03.631557 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc2edd37-18e4-4982-8758-c45199e53612","Type":"ContainerDied","Data":"ff2a520a5bc05f14a3950eb886abea3e6f45b094ecb18599cc384ef4c8e4be99"} Oct 30 18:36:04.639770 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:04.639729 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc2edd37-18e4-4982-8758-c45199e53612","Type":"ContainerStarted","Data":"5ba9af68b4b71aa97abe94670a74cdad43bd7ace34b0ef5302423f6f556e8253"} Oct 30 18:36:04.640220 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:04.639802 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc2edd37-18e4-4982-8758-c45199e53612","Type":"ContainerStarted","Data":"7aba5c2e04c1883af02c64480e25dd8696042752c0a55dabac116d41164b2776"} Oct 30 18:36:04.640220 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:04.639846 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc2edd37-18e4-4982-8758-c45199e53612","Type":"ContainerStarted","Data":"69bd64d419236b29d0e213bb9ef851dbd00a569d2bb69f4e616cf989f73d1f1d"} Oct 30 18:36:04.640220 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:04.639878 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc2edd37-18e4-4982-8758-c45199e53612","Type":"ContainerStarted","Data":"b0781fe9982d693595da9fdb4f2a49a7de7883f1a0e093a50b79b932b0516216"} Oct 30 18:36:04.640220 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:04.639903 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc2edd37-18e4-4982-8758-c45199e53612","Type":"ContainerStarted","Data":"fc74f0f2b2aa89221a2fe7baa8cc24cf2b9edd69c8e599d9aedc040e905870ab"} Oct 30 18:36:04.640220 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:04.639946 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc2edd37-18e4-4982-8758-c45199e53612","Type":"ContainerStarted","Data":"be6d5cf95704901fe08eda6a4320f6e4b2e5b89ce164e05ade50eb7311430caa"} Oct 30 18:36:04.771206 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:04.771119 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.749388839 podStartE2EDuration="3.771097236s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.708397ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:04.770173248 +0000 UTC m=+124.797278206" watchObservedRunningTime="2026-10-30 18:36:04.771097236 +0000 UTC m=+124.798202196" Oct 30 18:36:07.163979 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:36:07.163930 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:36:26.400175 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:36:26.400115 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:36:56.395931 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:36:56.395893 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:37:02.163243 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:37:02.163189 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:02.179463 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:37:02.179434 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:02.863800 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:37:02.863771 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:37:26.397744 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:37:26.397700 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:37:56.398200 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:37:56.398161 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:38:25.355807 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.355773 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw"] Oct 30 18:38:25.357793 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.357777 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" Oct 30 18:38:25.377986 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.377946 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cef05722-16ee-4c50-b69e-a8f54a70103e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f4db5dc9-zctbw\" (UID: \"cef05722-16ee-4c50-b69e-a8f54a70103e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" Oct 30 18:38:25.378154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.377995 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cef05722-16ee-4c50-b69e-a8f54a70103e-metrics-cert\") pod \"opendatahub-operator-controller-manager-67f4db5dc9-zctbw\" (UID: \"cef05722-16ee-4c50-b69e-a8f54a70103e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" Oct 30 18:38:25.378154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.378018 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cef05722-16ee-4c50-b69e-a8f54a70103e-manifests\") pod \"opendatahub-operator-controller-manager-67f4db5dc9-zctbw\" (UID: \"cef05722-16ee-4c50-b69e-a8f54a70103e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" Oct 30 18:38:25.378154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.378073 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cef05722-16ee-4c50-b69e-a8f54a70103e-tmp\") pod \"opendatahub-operator-controller-manager-67f4db5dc9-zctbw\" (UID: \"cef05722-16ee-4c50-b69e-a8f54a70103e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" Oct 30 18:38:25.378154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.378112 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cef05722-16ee-4c50-b69e-a8f54a70103e-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f4db5dc9-zctbw\" (UID: \"cef05722-16ee-4c50-b69e-a8f54a70103e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" Oct 30 18:38:25.378298 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.378217 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6gpl\" (UniqueName: \"kubernetes.io/projected/cef05722-16ee-4c50-b69e-a8f54a70103e-kube-api-access-k6gpl\") pod \"opendatahub-operator-controller-manager-67f4db5dc9-zctbw\" (UID: \"cef05722-16ee-4c50-b69e-a8f54a70103e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" Oct 30 18:38:25.410454 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.410426 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 18:38:25.410621 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.410579 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 18:38:25.459843 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.459811 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 18:38:25.459843 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.459835 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-xt8v5\"" Oct 30 18:38:25.460069 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.459813 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 18:38:25.460069 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.459916 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 18:38:25.479521 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.479482 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cef05722-16ee-4c50-b69e-a8f54a70103e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f4db5dc9-zctbw\" (UID: \"cef05722-16ee-4c50-b69e-a8f54a70103e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" Oct 30 18:38:25.479521 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.479522 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cef05722-16ee-4c50-b69e-a8f54a70103e-metrics-cert\") pod \"opendatahub-operator-controller-manager-67f4db5dc9-zctbw\" (UID: \"cef05722-16ee-4c50-b69e-a8f54a70103e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" Oct 30 18:38:25.479785 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.479540 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cef05722-16ee-4c50-b69e-a8f54a70103e-manifests\") pod \"opendatahub-operator-controller-manager-67f4db5dc9-zctbw\" (UID: \"cef05722-16ee-4c50-b69e-a8f54a70103e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" Oct 30 18:38:25.479785 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.479555 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cef05722-16ee-4c50-b69e-a8f54a70103e-tmp\") pod \"opendatahub-operator-controller-manager-67f4db5dc9-zctbw\" (UID: \"cef05722-16ee-4c50-b69e-a8f54a70103e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" Oct 30 18:38:25.479785 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.479570 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cef05722-16ee-4c50-b69e-a8f54a70103e-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f4db5dc9-zctbw\" (UID: \"cef05722-16ee-4c50-b69e-a8f54a70103e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" Oct 30 18:38:25.479785 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.479591 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6gpl\" (UniqueName: \"kubernetes.io/projected/cef05722-16ee-4c50-b69e-a8f54a70103e-kube-api-access-k6gpl\") pod \"opendatahub-operator-controller-manager-67f4db5dc9-zctbw\" (UID: \"cef05722-16ee-4c50-b69e-a8f54a70103e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" Oct 30 18:38:25.480060 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.480004 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cef05722-16ee-4c50-b69e-a8f54a70103e-manifests\") pod \"opendatahub-operator-controller-manager-67f4db5dc9-zctbw\" (UID: \"cef05722-16ee-4c50-b69e-a8f54a70103e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" Oct 30 18:38:25.482185 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.482162 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cef05722-16ee-4c50-b69e-a8f54a70103e-tmp\") pod \"opendatahub-operator-controller-manager-67f4db5dc9-zctbw\" (UID: \"cef05722-16ee-4c50-b69e-a8f54a70103e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" Oct 30 18:38:25.482584 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.482562 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cef05722-16ee-4c50-b69e-a8f54a70103e-metrics-cert\") pod \"opendatahub-operator-controller-manager-67f4db5dc9-zctbw\" (UID: \"cef05722-16ee-4c50-b69e-a8f54a70103e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" Oct 30 18:38:25.482627 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.482562 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cef05722-16ee-4c50-b69e-a8f54a70103e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f4db5dc9-zctbw\" (UID: \"cef05722-16ee-4c50-b69e-a8f54a70103e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" Oct 30 18:38:25.482627 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.482608 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cef05722-16ee-4c50-b69e-a8f54a70103e-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f4db5dc9-zctbw\" (UID: \"cef05722-16ee-4c50-b69e-a8f54a70103e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" Oct 30 18:38:25.498376 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.498341 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw"] Oct 30 18:38:25.542309 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.542271 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6gpl\" (UniqueName: \"kubernetes.io/projected/cef05722-16ee-4c50-b69e-a8f54a70103e-kube-api-access-k6gpl\") pod \"opendatahub-operator-controller-manager-67f4db5dc9-zctbw\" (UID: \"cef05722-16ee-4c50-b69e-a8f54a70103e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" Oct 30 18:38:25.671206 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:25.671089 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" Oct 30 18:38:26.178955 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:26.178918 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw"] Oct 30 18:38:26.182455 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:38:26.182421 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef05722_16ee_4c50_b69e_a8f54a70103e.slice/crio-a8803d803d5d99195ab374fa4b326ff9d90963efdc2b0590a0ff3a9cd5222919 WatchSource:0}: Error finding container a8803d803d5d99195ab374fa4b326ff9d90963efdc2b0590a0ff3a9cd5222919: Status 404 returned error can't find the container with id a8803d803d5d99195ab374fa4b326ff9d90963efdc2b0590a0ff3a9cd5222919 Oct 30 18:38:26.398107 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:38:26.398066 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:38:27.139000 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:27.138964 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" event={"ID":"cef05722-16ee-4c50-b69e-a8f54a70103e","Type":"ContainerStarted","Data":"a8803d803d5d99195ab374fa4b326ff9d90963efdc2b0590a0ff3a9cd5222919"} Oct 30 18:38:30.153797 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:30.153708 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" event={"ID":"cef05722-16ee-4c50-b69e-a8f54a70103e","Type":"ContainerStarted","Data":"c1f34cb6863b59cbff5e6f1cb8aed438c0579eaefd8275539236b43c3a9f7b23"} Oct 30 18:38:31.158600 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:31.158565 4531 generic.go:356] "Generic (PLEG): container finished" podID="cef05722-16ee-4c50-b69e-a8f54a70103e" containerID="c1f34cb6863b59cbff5e6f1cb8aed438c0579eaefd8275539236b43c3a9f7b23" exitCode=0 Oct 30 18:38:31.158992 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:31.158649 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" event={"ID":"cef05722-16ee-4c50-b69e-a8f54a70103e","Type":"ContainerDied","Data":"c1f34cb6863b59cbff5e6f1cb8aed438c0579eaefd8275539236b43c3a9f7b23"} Oct 30 18:38:32.164644 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:32.164606 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" event={"ID":"cef05722-16ee-4c50-b69e-a8f54a70103e","Type":"ContainerStarted","Data":"dd639ae7056d845ac37c472bf208a9677b4775ad6e77ecfd7fce65ecbb305909"} Oct 30 18:38:32.165066 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:32.164696 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" Oct 30 18:38:32.417789 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:32.417729 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" podStartSLOduration=2.411222596 podStartE2EDuration="7.417712696s" totalImagesPullingTime="4.517115212s" totalInitContainerRuntime="489.374888ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:32.367492007 +0000 UTC m=+272.394596965" watchObservedRunningTime="2026-10-30 18:38:32.417712696 +0000 UTC m=+272.444817655" Oct 30 18:38:37.471950 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:37.471913 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-wgz2m"] Oct 30 18:38:37.479852 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:37.479820 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wgz2m" Oct 30 18:38:37.535899 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:37.535827 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-lm7db\"" Oct 30 18:38:37.577270 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:37.577231 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a94d1b94-55b9-4ad7-95bb-a165c982eb3e-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-wgz2m\" (UID: \"a94d1b94-55b9-4ad7-95bb-a165c982eb3e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wgz2m" Oct 30 18:38:37.577270 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:37.577276 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9hz2\" (UniqueName: \"kubernetes.io/projected/a94d1b94-55b9-4ad7-95bb-a165c982eb3e-kube-api-access-v9hz2\") pod \"cert-manager-webhook-5dbbbf75cc-wgz2m\" (UID: \"a94d1b94-55b9-4ad7-95bb-a165c982eb3e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wgz2m" Oct 30 18:38:37.585151 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:37.585108 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 18:38:37.585151 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:37.585143 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 18:38:37.586180 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:37.586155 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-wgz2m"] Oct 30 18:38:37.678511 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:37.678465 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a94d1b94-55b9-4ad7-95bb-a165c982eb3e-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-wgz2m\" (UID: \"a94d1b94-55b9-4ad7-95bb-a165c982eb3e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wgz2m" Oct 30 18:38:37.678511 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:37.678512 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9hz2\" (UniqueName: \"kubernetes.io/projected/a94d1b94-55b9-4ad7-95bb-a165c982eb3e-kube-api-access-v9hz2\") pod \"cert-manager-webhook-5dbbbf75cc-wgz2m\" (UID: \"a94d1b94-55b9-4ad7-95bb-a165c982eb3e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wgz2m" Oct 30 18:38:37.743041 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:37.743007 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a94d1b94-55b9-4ad7-95bb-a165c982eb3e-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-wgz2m\" (UID: \"a94d1b94-55b9-4ad7-95bb-a165c982eb3e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wgz2m" Oct 30 18:38:37.743244 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:37.743061 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9hz2\" (UniqueName: \"kubernetes.io/projected/a94d1b94-55b9-4ad7-95bb-a165c982eb3e-kube-api-access-v9hz2\") pod \"cert-manager-webhook-5dbbbf75cc-wgz2m\" (UID: \"a94d1b94-55b9-4ad7-95bb-a165c982eb3e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wgz2m" Oct 30 18:38:37.810258 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:37.810223 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wgz2m" Oct 30 18:38:38.197832 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:38.197752 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-wgz2m"] Oct 30 18:38:38.201240 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:38:38.201206 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda94d1b94_55b9_4ad7_95bb_a165c982eb3e.slice/crio-10258d6b27b775fe96f27aa37a1997af06975da73c7250fee4646a473868e122 WatchSource:0}: Error finding container 10258d6b27b775fe96f27aa37a1997af06975da73c7250fee4646a473868e122: Status 404 returned error can't find the container with id 10258d6b27b775fe96f27aa37a1997af06975da73c7250fee4646a473868e122 Oct 30 18:38:39.191216 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:39.191169 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wgz2m" event={"ID":"a94d1b94-55b9-4ad7-95bb-a165c982eb3e","Type":"ContainerStarted","Data":"10258d6b27b775fe96f27aa37a1997af06975da73c7250fee4646a473868e122"} Oct 30 18:38:43.171241 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:43.171205 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67f4db5dc9-zctbw" Oct 30 18:38:43.208020 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:43.207987 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wgz2m" event={"ID":"a94d1b94-55b9-4ad7-95bb-a165c982eb3e","Type":"ContainerStarted","Data":"83700b836db24392b4eaefb7e43c51629aa84d769b231cc99778a6a0b62463fb"} Oct 30 18:38:43.208176 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:43.208033 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wgz2m" Oct 30 18:38:43.551221 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:43.551161 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wgz2m" podStartSLOduration=2.415146989 podStartE2EDuration="6.551119988s" totalImagesPullingTime="4.135972999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:43.550547781 +0000 UTC m=+283.577652731" watchObservedRunningTime="2026-10-30 18:38:43.551119988 +0000 UTC m=+283.578224947" Oct 30 18:38:44.971383 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:44.971348 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-r62sr"] Oct 30 18:38:44.973380 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:44.973352 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-r62sr" Oct 30 18:38:45.042166 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:45.042116 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77d4ff81-bf28-4c4a-9933-12562ddfaeb8-tmp\") pod \"jobset-operator-674cd59dc5-r62sr\" (UID: \"77d4ff81-bf28-4c4a-9933-12562ddfaeb8\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-r62sr" Oct 30 18:38:45.042362 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:45.042289 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srkps\" (UniqueName: \"kubernetes.io/projected/77d4ff81-bf28-4c4a-9933-12562ddfaeb8-kube-api-access-srkps\") pod \"jobset-operator-674cd59dc5-r62sr\" (UID: \"77d4ff81-bf28-4c4a-9933-12562ddfaeb8\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-r62sr" Oct 30 18:38:45.077338 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:45.077308 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 18:38:45.077338 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:45.077313 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-m2d2p\"" Oct 30 18:38:45.077338 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:45.077324 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:38:45.078501 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:45.078481 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-r62sr"] Oct 30 18:38:45.143408 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:45.143367 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77d4ff81-bf28-4c4a-9933-12562ddfaeb8-tmp\") pod \"jobset-operator-674cd59dc5-r62sr\" (UID: \"77d4ff81-bf28-4c4a-9933-12562ddfaeb8\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-r62sr" Oct 30 18:38:45.143618 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:45.143474 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srkps\" (UniqueName: \"kubernetes.io/projected/77d4ff81-bf28-4c4a-9933-12562ddfaeb8-kube-api-access-srkps\") pod \"jobset-operator-674cd59dc5-r62sr\" (UID: \"77d4ff81-bf28-4c4a-9933-12562ddfaeb8\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-r62sr" Oct 30 18:38:45.143830 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:45.143806 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77d4ff81-bf28-4c4a-9933-12562ddfaeb8-tmp\") pod \"jobset-operator-674cd59dc5-r62sr\" (UID: \"77d4ff81-bf28-4c4a-9933-12562ddfaeb8\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-r62sr" Oct 30 18:38:45.206525 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:45.206491 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srkps\" (UniqueName: \"kubernetes.io/projected/77d4ff81-bf28-4c4a-9933-12562ddfaeb8-kube-api-access-srkps\") pod \"jobset-operator-674cd59dc5-r62sr\" (UID: \"77d4ff81-bf28-4c4a-9933-12562ddfaeb8\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-r62sr" Oct 30 18:38:45.283534 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:45.283503 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-r62sr" Oct 30 18:38:45.653981 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:45.653896 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-r62sr"] Oct 30 18:38:45.657568 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:38:45.657535 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77d4ff81_bf28_4c4a_9933_12562ddfaeb8.slice/crio-0107bd668f72368d0bcebeb70fad9247ac4916334c1f529fd855133c4b7fdb1d WatchSource:0}: Error finding container 0107bd668f72368d0bcebeb70fad9247ac4916334c1f529fd855133c4b7fdb1d: Status 404 returned error can't find the container with id 0107bd668f72368d0bcebeb70fad9247ac4916334c1f529fd855133c4b7fdb1d Oct 30 18:38:46.220540 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:46.220503 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-r62sr" event={"ID":"77d4ff81-bf28-4c4a-9933-12562ddfaeb8","Type":"ContainerStarted","Data":"0107bd668f72368d0bcebeb70fad9247ac4916334c1f529fd855133c4b7fdb1d"} Oct 30 18:38:48.098518 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:48.098486 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-mgzs8"] Oct 30 18:38:48.100762 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:48.100737 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-mgzs8" Oct 30 18:38:48.167578 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:48.167540 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jzhp\" (UniqueName: \"kubernetes.io/projected/5dca4a65-543a-4414-9941-780a9f9b28ca-kube-api-access-9jzhp\") pod \"cert-manager-597b95cb84-mgzs8\" (UID: \"5dca4a65-543a-4414-9941-780a9f9b28ca\") " pod="cert-manager/cert-manager-597b95cb84-mgzs8" Oct 30 18:38:48.167758 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:48.167705 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5dca4a65-543a-4414-9941-780a9f9b28ca-bound-sa-token\") pod \"cert-manager-597b95cb84-mgzs8\" (UID: \"5dca4a65-543a-4414-9941-780a9f9b28ca\") " pod="cert-manager/cert-manager-597b95cb84-mgzs8" Oct 30 18:38:48.202860 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:48.202818 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-kz25x\"" Oct 30 18:38:48.205239 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:48.205210 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-mgzs8"] Oct 30 18:38:48.268856 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:48.268814 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jzhp\" (UniqueName: \"kubernetes.io/projected/5dca4a65-543a-4414-9941-780a9f9b28ca-kube-api-access-9jzhp\") pod \"cert-manager-597b95cb84-mgzs8\" (UID: \"5dca4a65-543a-4414-9941-780a9f9b28ca\") " pod="cert-manager/cert-manager-597b95cb84-mgzs8" Oct 30 18:38:48.269036 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:48.268904 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5dca4a65-543a-4414-9941-780a9f9b28ca-bound-sa-token\") pod \"cert-manager-597b95cb84-mgzs8\" (UID: \"5dca4a65-543a-4414-9941-780a9f9b28ca\") " pod="cert-manager/cert-manager-597b95cb84-mgzs8" Oct 30 18:38:48.330386 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:48.330355 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5dca4a65-543a-4414-9941-780a9f9b28ca-bound-sa-token\") pod \"cert-manager-597b95cb84-mgzs8\" (UID: \"5dca4a65-543a-4414-9941-780a9f9b28ca\") " pod="cert-manager/cert-manager-597b95cb84-mgzs8" Oct 30 18:38:48.331250 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:48.331199 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jzhp\" (UniqueName: \"kubernetes.io/projected/5dca4a65-543a-4414-9941-780a9f9b28ca-kube-api-access-9jzhp\") pod \"cert-manager-597b95cb84-mgzs8\" (UID: \"5dca4a65-543a-4414-9941-780a9f9b28ca\") " pod="cert-manager/cert-manager-597b95cb84-mgzs8" Oct 30 18:38:48.415928 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:48.415824 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-mgzs8" Oct 30 18:38:48.806965 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:48.806356 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-mgzs8"] Oct 30 18:38:48.818809 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:38:48.818775 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5dca4a65_543a_4414_9941_780a9f9b28ca.slice/crio-5096c7ff0a209b1019c38363e5038fe3688ff184c6f81fc4dc63f5137db086aa WatchSource:0}: Error finding container 5096c7ff0a209b1019c38363e5038fe3688ff184c6f81fc4dc63f5137db086aa: Status 404 returned error can't find the container with id 5096c7ff0a209b1019c38363e5038fe3688ff184c6f81fc4dc63f5137db086aa Oct 30 18:38:49.214704 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:49.214626 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wgz2m" Oct 30 18:38:49.243301 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:49.243265 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-mgzs8" event={"ID":"5dca4a65-543a-4414-9941-780a9f9b28ca","Type":"ContainerStarted","Data":"bb07272912a307e3eed98db22d60fd0bfa54703b5ac21fb139be26c8a5c0477b"} Oct 30 18:38:49.243467 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:49.243317 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-mgzs8" event={"ID":"5dca4a65-543a-4414-9941-780a9f9b28ca","Type":"ContainerStarted","Data":"5096c7ff0a209b1019c38363e5038fe3688ff184c6f81fc4dc63f5137db086aa"} Oct 30 18:38:49.482321 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:49.481203 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-mgzs8" podStartSLOduration=1.481182435 podStartE2EDuration="1.481182435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:49.480453309 +0000 UTC m=+289.507558271" watchObservedRunningTime="2026-10-30 18:38:49.481182435 +0000 UTC m=+289.508287388" Oct 30 18:38:50.248984 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:50.248936 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-r62sr" event={"ID":"77d4ff81-bf28-4c4a-9933-12562ddfaeb8","Type":"ContainerStarted","Data":"71feae093e1e151a2e6c03e386e3df4cc0ee6dabbb0f181706756abebc24a1b6"} Oct 30 18:38:50.368138 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:38:50.368083 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-r62sr" podStartSLOduration=2.089294804 podStartE2EDuration="6.368069084s" totalImagesPullingTime="4.27877428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:50.367813556 +0000 UTC m=+290.394918516" watchObservedRunningTime="2026-10-30 18:38:50.368069084 +0000 UTC m=+290.395174043" Oct 30 18:38:56.395667 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:38:56.395626 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:39:01.024812 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:01.024775 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-9468cdd5d-vhp9x"] Oct 30 18:39:01.029784 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:01.029763 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9468cdd5d-vhp9x" Oct 30 18:39:01.082422 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:01.082385 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 18:39:01.132081 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:01.132041 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 18:39:01.132081 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:01.132053 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 18:39:01.132081 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:01.132069 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 18:39:01.132495 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:01.132164 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8m6s4\"" Oct 30 18:39:01.134028 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:01.134005 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9468cdd5d-vhp9x"] Oct 30 18:39:01.184937 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:01.184907 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2650a306-fa87-40d3-b893-5480a959ac51-tls-certs\") pod \"kube-auth-proxy-9468cdd5d-vhp9x\" (UID: \"2650a306-fa87-40d3-b893-5480a959ac51\") " pod="openshift-ingress/kube-auth-proxy-9468cdd5d-vhp9x" Oct 30 18:39:01.185080 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:01.184949 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c8rq\" (UniqueName: \"kubernetes.io/projected/2650a306-fa87-40d3-b893-5480a959ac51-kube-api-access-5c8rq\") pod \"kube-auth-proxy-9468cdd5d-vhp9x\" (UID: \"2650a306-fa87-40d3-b893-5480a959ac51\") " pod="openshift-ingress/kube-auth-proxy-9468cdd5d-vhp9x" Oct 30 18:39:01.185080 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:01.184982 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2650a306-fa87-40d3-b893-5480a959ac51-tmp\") pod \"kube-auth-proxy-9468cdd5d-vhp9x\" (UID: \"2650a306-fa87-40d3-b893-5480a959ac51\") " pod="openshift-ingress/kube-auth-proxy-9468cdd5d-vhp9x" Oct 30 18:39:01.286488 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:01.286413 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2650a306-fa87-40d3-b893-5480a959ac51-tls-certs\") pod \"kube-auth-proxy-9468cdd5d-vhp9x\" (UID: \"2650a306-fa87-40d3-b893-5480a959ac51\") " pod="openshift-ingress/kube-auth-proxy-9468cdd5d-vhp9x" Oct 30 18:39:01.286488 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:01.286457 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c8rq\" (UniqueName: \"kubernetes.io/projected/2650a306-fa87-40d3-b893-5480a959ac51-kube-api-access-5c8rq\") pod \"kube-auth-proxy-9468cdd5d-vhp9x\" (UID: \"2650a306-fa87-40d3-b893-5480a959ac51\") " pod="openshift-ingress/kube-auth-proxy-9468cdd5d-vhp9x" Oct 30 18:39:01.286687 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:01.286491 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2650a306-fa87-40d3-b893-5480a959ac51-tmp\") pod \"kube-auth-proxy-9468cdd5d-vhp9x\" (UID: \"2650a306-fa87-40d3-b893-5480a959ac51\") " pod="openshift-ingress/kube-auth-proxy-9468cdd5d-vhp9x" Oct 30 18:39:01.288802 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:01.288773 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2650a306-fa87-40d3-b893-5480a959ac51-tmp\") pod \"kube-auth-proxy-9468cdd5d-vhp9x\" (UID: \"2650a306-fa87-40d3-b893-5480a959ac51\") " pod="openshift-ingress/kube-auth-proxy-9468cdd5d-vhp9x" Oct 30 18:39:01.289168 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:01.289121 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2650a306-fa87-40d3-b893-5480a959ac51-tls-certs\") pod \"kube-auth-proxy-9468cdd5d-vhp9x\" (UID: \"2650a306-fa87-40d3-b893-5480a959ac51\") " pod="openshift-ingress/kube-auth-proxy-9468cdd5d-vhp9x" Oct 30 18:39:01.348972 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:01.348937 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c8rq\" (UniqueName: \"kubernetes.io/projected/2650a306-fa87-40d3-b893-5480a959ac51-kube-api-access-5c8rq\") pod \"kube-auth-proxy-9468cdd5d-vhp9x\" (UID: \"2650a306-fa87-40d3-b893-5480a959ac51\") " pod="openshift-ingress/kube-auth-proxy-9468cdd5d-vhp9x" Oct 30 18:39:01.642798 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:01.642723 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9468cdd5d-vhp9x" Oct 30 18:39:02.013215 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:02.013177 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9468cdd5d-vhp9x"] Oct 30 18:39:02.016516 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:39:02.016488 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2650a306_fa87_40d3_b893_5480a959ac51.slice/crio-f8e45ce5b7fe7c06ff79dc01070111efc5407593eb5041d89929063e743544bf WatchSource:0}: Error finding container f8e45ce5b7fe7c06ff79dc01070111efc5407593eb5041d89929063e743544bf: Status 404 returned error can't find the container with id f8e45ce5b7fe7c06ff79dc01070111efc5407593eb5041d89929063e743544bf Oct 30 18:39:02.018484 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:02.018465 4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:39:02.297450 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:02.297368 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9468cdd5d-vhp9x" event={"ID":"2650a306-fa87-40d3-b893-5480a959ac51","Type":"ContainerStarted","Data":"f8e45ce5b7fe7c06ff79dc01070111efc5407593eb5041d89929063e743544bf"} Oct 30 18:39:05.312509 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:05.312466 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9468cdd5d-vhp9x" event={"ID":"2650a306-fa87-40d3-b893-5480a959ac51","Type":"ContainerStarted","Data":"eec6dfc0dfe06f4cd3cd258081c3a8df6032df75a6f8d9f09a194f109f7b1653"} Oct 30 18:39:08.755976 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:08.755905 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-9468cdd5d-vhp9x" podStartSLOduration=5.86977267 podStartE2EDuration="8.755887986s" totalImagesPullingTime="2.886115316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:05.432800651 +0000 UTC m=+305.459905589" watchObservedRunningTime="2026-10-30 18:39:08.755887986 +0000 UTC m=+308.782992963" Oct 30 18:39:08.756422 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:08.756230 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-rkjnb"] Oct 30 18:39:08.759721 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:08.759696 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rkjnb" Oct 30 18:39:08.821200 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:08.821168 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-rkjnb"] Oct 30 18:39:08.822372 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:08.822345 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 18:39:08.822507 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:08.822388 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 18:39:08.822507 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:08.822490 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 18:39:08.824855 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:08.824831 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4m2vx\"" Oct 30 18:39:08.824987 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:08.824913 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 18:39:08.853392 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:08.853345 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twn7b\" (UniqueName: \"kubernetes.io/projected/0c895121-1801-4d2a-a4d2-c33e167ec4b5-kube-api-access-twn7b\") pod \"kuadrant-console-plugin-66c4fb575d-rkjnb\" (UID: \"0c895121-1801-4d2a-a4d2-c33e167ec4b5\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rkjnb" Oct 30 18:39:08.853579 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:08.853403 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0c895121-1801-4d2a-a4d2-c33e167ec4b5-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-rkjnb\" (UID: \"0c895121-1801-4d2a-a4d2-c33e167ec4b5\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rkjnb" Oct 30 18:39:08.853579 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:08.853516 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c895121-1801-4d2a-a4d2-c33e167ec4b5-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-rkjnb\" (UID: \"0c895121-1801-4d2a-a4d2-c33e167ec4b5\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rkjnb" Oct 30 18:39:08.954633 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:08.954597 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0c895121-1801-4d2a-a4d2-c33e167ec4b5-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-rkjnb\" (UID: \"0c895121-1801-4d2a-a4d2-c33e167ec4b5\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rkjnb" Oct 30 18:39:08.954826 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:08.954651 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c895121-1801-4d2a-a4d2-c33e167ec4b5-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-rkjnb\" (UID: \"0c895121-1801-4d2a-a4d2-c33e167ec4b5\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rkjnb" Oct 30 18:39:08.954826 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:08.954683 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twn7b\" (UniqueName: \"kubernetes.io/projected/0c895121-1801-4d2a-a4d2-c33e167ec4b5-kube-api-access-twn7b\") pod \"kuadrant-console-plugin-66c4fb575d-rkjnb\" (UID: \"0c895121-1801-4d2a-a4d2-c33e167ec4b5\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rkjnb" Oct 30 18:39:08.955322 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:08.955301 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0c895121-1801-4d2a-a4d2-c33e167ec4b5-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-rkjnb\" (UID: \"0c895121-1801-4d2a-a4d2-c33e167ec4b5\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rkjnb" Oct 30 18:39:08.957229 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:08.957204 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c895121-1801-4d2a-a4d2-c33e167ec4b5-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-rkjnb\" (UID: \"0c895121-1801-4d2a-a4d2-c33e167ec4b5\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rkjnb" Oct 30 18:39:09.023605 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:09.023541 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twn7b\" (UniqueName: \"kubernetes.io/projected/0c895121-1801-4d2a-a4d2-c33e167ec4b5-kube-api-access-twn7b\") pod \"kuadrant-console-plugin-66c4fb575d-rkjnb\" (UID: \"0c895121-1801-4d2a-a4d2-c33e167ec4b5\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rkjnb" Oct 30 18:39:09.072719 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:09.072656 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rkjnb" Oct 30 18:39:09.376172 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:09.376042 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5f854c649f-r8lsd"] Oct 30 18:39:09.380637 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:09.380600 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5f854c649f-r8lsd" Oct 30 18:39:09.439590 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:09.439522 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-5z6gt\"" Oct 30 18:39:09.439805 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:09.439607 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 18:39:09.439805 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:09.439780 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 18:39:09.443725 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:09.443697 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5f854c649f-r8lsd"] Oct 30 18:39:09.453324 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:09.453301 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 18:39:09.487614 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:09.487437 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-rkjnb"] Oct 30 18:39:09.490313 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:39:09.490276 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c895121_1801_4d2a_a4d2_c33e167ec4b5.slice/crio-d31ec42d74826f70d778a1ccc23e5a06356197732f702bc5b06463d5628c3451 WatchSource:0}: Error finding container d31ec42d74826f70d778a1ccc23e5a06356197732f702bc5b06463d5628c3451: Status 404 returned error can't find the container with id d31ec42d74826f70d778a1ccc23e5a06356197732f702bc5b06463d5628c3451 Oct 30 18:39:09.560658 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:09.560625 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dn54\" (UniqueName: \"kubernetes.io/projected/4c587a91-2546-4d19-8c67-efa1b06f187d-kube-api-access-4dn54\") pod \"jobset-controller-manager-5f854c649f-r8lsd\" (UID: \"4c587a91-2546-4d19-8c67-efa1b06f187d\") " pod="openshift-jobset-operator/jobset-controller-manager-5f854c649f-r8lsd" Oct 30 18:39:09.560928 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:09.560878 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4c587a91-2546-4d19-8c67-efa1b06f187d-manager-config\") pod \"jobset-controller-manager-5f854c649f-r8lsd\" (UID: \"4c587a91-2546-4d19-8c67-efa1b06f187d\") " pod="openshift-jobset-operator/jobset-controller-manager-5f854c649f-r8lsd" Oct 30 18:39:09.561087 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:09.560951 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c587a91-2546-4d19-8c67-efa1b06f187d-metrics-certs\") pod \"jobset-controller-manager-5f854c649f-r8lsd\" (UID: \"4c587a91-2546-4d19-8c67-efa1b06f187d\") " pod="openshift-jobset-operator/jobset-controller-manager-5f854c649f-r8lsd" Oct 30 18:39:09.561087 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:09.560979 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c587a91-2546-4d19-8c67-efa1b06f187d-cert\") pod \"jobset-controller-manager-5f854c649f-r8lsd\" (UID: \"4c587a91-2546-4d19-8c67-efa1b06f187d\") " pod="openshift-jobset-operator/jobset-controller-manager-5f854c649f-r8lsd" Oct 30 18:39:09.662237 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:09.662168 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4c587a91-2546-4d19-8c67-efa1b06f187d-manager-config\") pod \"jobset-controller-manager-5f854c649f-r8lsd\" (UID: \"4c587a91-2546-4d19-8c67-efa1b06f187d\") " pod="openshift-jobset-operator/jobset-controller-manager-5f854c649f-r8lsd" Oct 30 18:39:09.662237 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:09.662218 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c587a91-2546-4d19-8c67-efa1b06f187d-metrics-certs\") pod \"jobset-controller-manager-5f854c649f-r8lsd\" (UID: \"4c587a91-2546-4d19-8c67-efa1b06f187d\") " pod="openshift-jobset-operator/jobset-controller-manager-5f854c649f-r8lsd" Oct 30 18:39:09.662237 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:09.662237 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c587a91-2546-4d19-8c67-efa1b06f187d-cert\") pod \"jobset-controller-manager-5f854c649f-r8lsd\" (UID: \"4c587a91-2546-4d19-8c67-efa1b06f187d\") " pod="openshift-jobset-operator/jobset-controller-manager-5f854c649f-r8lsd" Oct 30 18:39:09.662433 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:09.662291 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dn54\" (UniqueName: \"kubernetes.io/projected/4c587a91-2546-4d19-8c67-efa1b06f187d-kube-api-access-4dn54\") pod \"jobset-controller-manager-5f854c649f-r8lsd\" (UID: \"4c587a91-2546-4d19-8c67-efa1b06f187d\") " pod="openshift-jobset-operator/jobset-controller-manager-5f854c649f-r8lsd" Oct 30 18:39:09.662817 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:09.662788 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4c587a91-2546-4d19-8c67-efa1b06f187d-manager-config\") pod \"jobset-controller-manager-5f854c649f-r8lsd\" (UID: \"4c587a91-2546-4d19-8c67-efa1b06f187d\") " pod="openshift-jobset-operator/jobset-controller-manager-5f854c649f-r8lsd" Oct 30 18:39:09.664762 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:09.664738 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c587a91-2546-4d19-8c67-efa1b06f187d-cert\") pod \"jobset-controller-manager-5f854c649f-r8lsd\" (UID: \"4c587a91-2546-4d19-8c67-efa1b06f187d\") " pod="openshift-jobset-operator/jobset-controller-manager-5f854c649f-r8lsd" Oct 30 18:39:09.664843 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:09.664787 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c587a91-2546-4d19-8c67-efa1b06f187d-metrics-certs\") pod \"jobset-controller-manager-5f854c649f-r8lsd\" (UID: \"4c587a91-2546-4d19-8c67-efa1b06f187d\") " pod="openshift-jobset-operator/jobset-controller-manager-5f854c649f-r8lsd" Oct 30 18:39:09.723760 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:09.723729 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dn54\" (UniqueName: \"kubernetes.io/projected/4c587a91-2546-4d19-8c67-efa1b06f187d-kube-api-access-4dn54\") pod \"jobset-controller-manager-5f854c649f-r8lsd\" (UID: \"4c587a91-2546-4d19-8c67-efa1b06f187d\") " pod="openshift-jobset-operator/jobset-controller-manager-5f854c649f-r8lsd" Oct 30 18:39:09.993864 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:09.993830 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5f854c649f-r8lsd" Oct 30 18:39:10.334470 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:10.334392 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rkjnb" event={"ID":"0c895121-1801-4d2a-a4d2-c33e167ec4b5","Type":"ContainerStarted","Data":"d31ec42d74826f70d778a1ccc23e5a06356197732f702bc5b06463d5628c3451"} Oct 30 18:39:10.417024 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:10.416993 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5f854c649f-r8lsd"] Oct 30 18:39:10.418797 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:39:10.418770 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c587a91_2546_4d19_8c67_efa1b06f187d.slice/crio-82222a9a99255497488f95e5ddd12ebfe5423fe5c24cf67344d3c13522842bd3 WatchSource:0}: Error finding container 82222a9a99255497488f95e5ddd12ebfe5423fe5c24cf67344d3c13522842bd3: Status 404 returned error can't find the container with id 82222a9a99255497488f95e5ddd12ebfe5423fe5c24cf67344d3c13522842bd3 Oct 30 18:39:11.341959 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:11.341858 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5f854c649f-r8lsd" event={"ID":"4c587a91-2546-4d19-8c67-efa1b06f187d","Type":"ContainerStarted","Data":"82222a9a99255497488f95e5ddd12ebfe5423fe5c24cf67344d3c13522842bd3"} Oct 30 18:39:15.363873 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:15.363832 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5f854c649f-r8lsd" event={"ID":"4c587a91-2546-4d19-8c67-efa1b06f187d","Type":"ContainerStarted","Data":"a076bf00f6652ba4dedf1cf8278900fe130361824d3571d315c2ea33bc43ebf0"} Oct 30 18:39:15.364459 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:15.363977 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5f854c649f-r8lsd" Oct 30 18:39:15.488209 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:15.487998 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5f854c649f-r8lsd" podStartSLOduration=2.007706231 podStartE2EDuration="6.487978444s" totalImagesPullingTime="4.480272213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:15.486223185 +0000 UTC m=+315.513328145" watchObservedRunningTime="2026-10-30 18:39:15.487978444 +0000 UTC m=+315.515083407" Oct 30 18:39:16.369618 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:16.369583 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rkjnb" event={"ID":"0c895121-1801-4d2a-a4d2-c33e167ec4b5","Type":"ContainerStarted","Data":"1de4a57352d1f311c881202690072187a4de4edcb3f0481fb088d3fb7c1154e2"} Oct 30 18:39:16.502917 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:16.502855 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rkjnb" podStartSLOduration=2.318602641 podStartE2EDuration="8.502838978s" totalImagesPullingTime="6.184236337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:16.502215546 +0000 UTC m=+316.529320506" watchObservedRunningTime="2026-10-30 18:39:16.502838978 +0000 UTC m=+316.529943938" Oct 30 18:39:26.374861 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:39:26.374831 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5f854c649f-r8lsd" Oct 30 18:39:26.396652 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:39:26.396621 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:39:56.399331 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:39:56.399291 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:40:20.549049 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:40:20.549012 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2w5pv"] Oct 30 18:40:20.551294 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:40:20.551268 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2w5pv" Oct 30 18:40:20.576490 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:40:20.576446 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ee2539c-afb2-4060-8461-61c4471b31d0-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-2w5pv\" (UID: \"4ee2539c-afb2-4060-8461-61c4471b31d0\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2w5pv" Oct 30 18:40:20.576490 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:40:20.576491 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj6tw\" (UniqueName: \"kubernetes.io/projected/4ee2539c-afb2-4060-8461-61c4471b31d0-kube-api-access-qj6tw\") pod \"llmisvc-controller-manager-869d5dcc5d-2w5pv\" (UID: \"4ee2539c-afb2-4060-8461-61c4471b31d0\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2w5pv" Oct 30 18:40:20.660879 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:40:20.660846 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-jxxrr\"" Oct 30 18:40:20.661110 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:40:20.660993 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 18:40:20.661110 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:40:20.660998 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 18:40:20.661110 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:40:20.661041 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 18:40:20.662637 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:40:20.662609 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2w5pv"] Oct 30 18:40:20.677755 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:40:20.677716 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ee2539c-afb2-4060-8461-61c4471b31d0-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-2w5pv\" (UID: \"4ee2539c-afb2-4060-8461-61c4471b31d0\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2w5pv" Oct 30 18:40:20.677958 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:40:20.677767 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj6tw\" (UniqueName: \"kubernetes.io/projected/4ee2539c-afb2-4060-8461-61c4471b31d0-kube-api-access-qj6tw\") pod \"llmisvc-controller-manager-869d5dcc5d-2w5pv\" (UID: \"4ee2539c-afb2-4060-8461-61c4471b31d0\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2w5pv" Oct 30 18:40:20.680730 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:40:20.680676 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ee2539c-afb2-4060-8461-61c4471b31d0-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-2w5pv\" (UID: \"4ee2539c-afb2-4060-8461-61c4471b31d0\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2w5pv" Oct 30 18:40:20.751953 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:40:20.751925 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj6tw\" (UniqueName: \"kubernetes.io/projected/4ee2539c-afb2-4060-8461-61c4471b31d0-kube-api-access-qj6tw\") pod \"llmisvc-controller-manager-869d5dcc5d-2w5pv\" (UID: \"4ee2539c-afb2-4060-8461-61c4471b31d0\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2w5pv" Oct 30 18:40:20.863249 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:40:20.863157 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2w5pv" Oct 30 18:40:21.238381 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:40:21.238290 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2w5pv"] Oct 30 18:40:21.241754 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:40:21.241718 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4ee2539c_afb2_4060_8461_61c4471b31d0.slice/crio-48af93ac487532e158bc3525d03aac0891b3875066f5071fda0fc98575145f45 WatchSource:0}: Error finding container 48af93ac487532e158bc3525d03aac0891b3875066f5071fda0fc98575145f45: Status 404 returned error can't find the container with id 48af93ac487532e158bc3525d03aac0891b3875066f5071fda0fc98575145f45 Oct 30 18:40:21.639650 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:40:21.639610 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2w5pv" event={"ID":"4ee2539c-afb2-4060-8461-61c4471b31d0","Type":"ContainerStarted","Data":"48af93ac487532e158bc3525d03aac0891b3875066f5071fda0fc98575145f45"} Oct 30 18:40:24.659669 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:40:24.659623 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2w5pv" event={"ID":"4ee2539c-afb2-4060-8461-61c4471b31d0","Type":"ContainerStarted","Data":"a09d0891ea97a64e85f712f5e7b3f440f05fd96f5fc067b92ef7f6b671488b16"} Oct 30 18:40:24.660185 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:40:24.659728 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2w5pv" Oct 30 18:40:24.780524 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:40:24.780470 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2w5pv" podStartSLOduration=2.003465715 podStartE2EDuration="4.780450737s" totalImagesPullingTime="2.776985022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:24.779077666 +0000 UTC m=+384.806182622" watchObservedRunningTime="2026-10-30 18:40:24.780450737 +0000 UTC m=+384.807555695" Oct 30 18:40:26.400177 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:40:26.400119 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:40:55.666112 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:40:55.666080 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2w5pv" Oct 30 18:40:56.398506 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:40:56.398464 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:41:26.399302 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:41:26.399255 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:41:39.732287 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:41:39.732242 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-56jrt"] Oct 30 18:41:39.735634 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:41:39.735612 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-56jrt" Oct 30 18:41:39.788455 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:41:39.788411 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a47b1484-d120-4c8e-ad86-1309eb472e84-cert\") pod \"llmisvc-controller-manager-6466c554d8-56jrt\" (UID: \"a47b1484-d120-4c8e-ad86-1309eb472e84\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-56jrt" Oct 30 18:41:39.788643 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:41:39.788469 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8stcv\" (UniqueName: \"kubernetes.io/projected/a47b1484-d120-4c8e-ad86-1309eb472e84-kube-api-access-8stcv\") pod \"llmisvc-controller-manager-6466c554d8-56jrt\" (UID: \"a47b1484-d120-4c8e-ad86-1309eb472e84\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-56jrt" Oct 30 18:41:39.795208 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:41:39.795168 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-56jrt"] Oct 30 18:41:39.889776 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:41:39.889738 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a47b1484-d120-4c8e-ad86-1309eb472e84-cert\") pod \"llmisvc-controller-manager-6466c554d8-56jrt\" (UID: \"a47b1484-d120-4c8e-ad86-1309eb472e84\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-56jrt" Oct 30 18:41:39.889963 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:41:39.889794 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8stcv\" (UniqueName: \"kubernetes.io/projected/a47b1484-d120-4c8e-ad86-1309eb472e84-kube-api-access-8stcv\") pod \"llmisvc-controller-manager-6466c554d8-56jrt\" (UID: \"a47b1484-d120-4c8e-ad86-1309eb472e84\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-56jrt" Oct 30 18:41:39.892368 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:41:39.892340 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a47b1484-d120-4c8e-ad86-1309eb472e84-cert\") pod \"llmisvc-controller-manager-6466c554d8-56jrt\" (UID: \"a47b1484-d120-4c8e-ad86-1309eb472e84\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-56jrt" Oct 30 18:41:39.951287 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:41:39.951250 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8stcv\" (UniqueName: \"kubernetes.io/projected/a47b1484-d120-4c8e-ad86-1309eb472e84-kube-api-access-8stcv\") pod \"llmisvc-controller-manager-6466c554d8-56jrt\" (UID: \"a47b1484-d120-4c8e-ad86-1309eb472e84\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-56jrt" Oct 30 18:41:40.048682 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:41:40.048645 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-56jrt" Oct 30 18:41:40.422595 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:41:40.422511 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-56jrt"] Oct 30 18:41:40.426077 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:41:40.426051 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47b1484_d120_4c8e_ad86_1309eb472e84.slice/crio-031eaf0fd85004435ca715d6ce0c4c003d5e19640aef747ccdd4b7acef24a373 WatchSource:0}: Error finding container 031eaf0fd85004435ca715d6ce0c4c003d5e19640aef747ccdd4b7acef24a373: Status 404 returned error can't find the container with id 031eaf0fd85004435ca715d6ce0c4c003d5e19640aef747ccdd4b7acef24a373 Oct 30 18:41:40.987902 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:41:40.987869 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-56jrt" event={"ID":"a47b1484-d120-4c8e-ad86-1309eb472e84","Type":"ContainerStarted","Data":"031eaf0fd85004435ca715d6ce0c4c003d5e19640aef747ccdd4b7acef24a373"} Oct 30 18:41:41.993976 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:41:41.993937 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-56jrt" event={"ID":"a47b1484-d120-4c8e-ad86-1309eb472e84","Type":"ContainerStarted","Data":"b3cf22811d02bafc52c5dfdd3de989337985dd435937528bd94161ec6ca5a6d0"} Oct 30 18:41:41.994416 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:41:41.994079 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-56jrt" Oct 30 18:41:42.110281 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:41:42.110228 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-56jrt" podStartSLOduration=2.282267655 podStartE2EDuration="3.110212501s" totalImagesPullingTime="827.944846ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:42.109753574 +0000 UTC m=+462.136858535" watchObservedRunningTime="2026-10-30 18:41:42.110212501 +0000 UTC m=+462.137317460" Oct 30 18:41:56.398622 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:41:56.398580 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:42:13.000301 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:42:13.000218 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-56jrt" Oct 30 18:42:13.166020 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:42:13.165988 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2w5pv"] Oct 30 18:42:13.166282 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:42:13.166257 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2w5pv" podUID="4ee2539c-afb2-4060-8461-61c4471b31d0" containerName="manager" containerID="cri-o://a09d0891ea97a64e85f712f5e7b3f440f05fd96f5fc067b92ef7f6b671488b16" gracePeriod=30 Oct 30 18:42:13.428021 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:42:13.427990 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2w5pv" Oct 30 18:42:13.577176 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:42:13.577058 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ee2539c-afb2-4060-8461-61c4471b31d0-kube-api-access-qj6tw\" (UniqueName: \"kubernetes.io/projected/4ee2539c-afb2-4060-8461-61c4471b31d0-kube-api-access-qj6tw\") pod \"4ee2539c-afb2-4060-8461-61c4471b31d0\" (UID: \"4ee2539c-afb2-4060-8461-61c4471b31d0\") " Oct 30 18:42:13.577333 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:42:13.577186 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ee2539c-afb2-4060-8461-61c4471b31d0-cert\" (UniqueName: \"kubernetes.io/secret/4ee2539c-afb2-4060-8461-61c4471b31d0-cert\") pod \"4ee2539c-afb2-4060-8461-61c4471b31d0\" (UID: \"4ee2539c-afb2-4060-8461-61c4471b31d0\") " Oct 30 18:42:13.579437 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:42:13.579407 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ee2539c-afb2-4060-8461-61c4471b31d0-cert" pod "4ee2539c-afb2-4060-8461-61c4471b31d0" (UID: "4ee2539c-afb2-4060-8461-61c4471b31d0"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:42:13.579558 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:42:13.579522 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee2539c-afb2-4060-8461-61c4471b31d0-kube-api-access-qj6tw" pod "4ee2539c-afb2-4060-8461-61c4471b31d0" (UID: "4ee2539c-afb2-4060-8461-61c4471b31d0"). InnerVolumeSpecName "kube-api-access-qj6tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:13.678440 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:42:13.678404 4531 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ee2539c-afb2-4060-8461-61c4471b31d0-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:42:13.678440 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:42:13.678433 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj6tw\" (UniqueName: \"kubernetes.io/projected/4ee2539c-afb2-4060-8461-61c4471b31d0-kube-api-access-qj6tw\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:42:14.129729 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:42:14.129692 4531 generic.go:356] "Generic (PLEG): container finished" podID="4ee2539c-afb2-4060-8461-61c4471b31d0" containerID="a09d0891ea97a64e85f712f5e7b3f440f05fd96f5fc067b92ef7f6b671488b16" exitCode=0 Oct 30 18:42:14.130215 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:42:14.129748 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2w5pv" Oct 30 18:42:14.130215 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:42:14.129775 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2w5pv" event={"ID":"4ee2539c-afb2-4060-8461-61c4471b31d0","Type":"ContainerDied","Data":"a09d0891ea97a64e85f712f5e7b3f440f05fd96f5fc067b92ef7f6b671488b16"} Oct 30 18:42:14.130215 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:42:14.129836 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2w5pv" event={"ID":"4ee2539c-afb2-4060-8461-61c4471b31d0","Type":"ContainerDied","Data":"48af93ac487532e158bc3525d03aac0891b3875066f5071fda0fc98575145f45"} Oct 30 18:42:14.130215 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:42:14.129860 4531 scope.go:122] "RemoveContainer" containerID="a09d0891ea97a64e85f712f5e7b3f440f05fd96f5fc067b92ef7f6b671488b16" Oct 30 18:42:14.138832 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:42:14.138808 4531 scope.go:122] "RemoveContainer" containerID="a09d0891ea97a64e85f712f5e7b3f440f05fd96f5fc067b92ef7f6b671488b16" Oct 30 18:42:14.139239 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:42:14.139209 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a09d0891ea97a64e85f712f5e7b3f440f05fd96f5fc067b92ef7f6b671488b16\": container with ID starting with a09d0891ea97a64e85f712f5e7b3f440f05fd96f5fc067b92ef7f6b671488b16 not found: ID does not exist" containerID="a09d0891ea97a64e85f712f5e7b3f440f05fd96f5fc067b92ef7f6b671488b16" Oct 30 18:42:14.139327 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:42:14.139251 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a09d0891ea97a64e85f712f5e7b3f440f05fd96f5fc067b92ef7f6b671488b16"} err="failed to get container status \"a09d0891ea97a64e85f712f5e7b3f440f05fd96f5fc067b92ef7f6b671488b16\": rpc error: code = NotFound desc = could not find container \"a09d0891ea97a64e85f712f5e7b3f440f05fd96f5fc067b92ef7f6b671488b16\": container with ID starting with a09d0891ea97a64e85f712f5e7b3f440f05fd96f5fc067b92ef7f6b671488b16 not found: ID does not exist" Oct 30 18:42:14.293043 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:42:14.293005 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2w5pv"] Oct 30 18:42:14.342374 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:42:14.342336 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2w5pv"] Oct 30 18:42:16.097371 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:42:16.097325 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ee2539c-afb2-4060-8461-61c4471b31d0" path="/var/lib/kubelet/pods/4ee2539c-afb2-4060-8461-61c4471b31d0/volumes" Oct 30 18:42:26.399472 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:42:26.399432 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:42:56.398395 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:42:56.398335 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:43:26.400208 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:43:26.400168 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:43:51.080157 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.080046 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd"] Oct 30 18:43:51.080853 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.080398 4531 cpu_manager.go:414] "containerMap: removing container" podUID="4ee2539c-afb2-4060-8461-61c4471b31d0" containerName="manager" Oct 30 18:43:51.080853 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.080409 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ee2539c-afb2-4060-8461-61c4471b31d0" containerName="manager" Oct 30 18:43:51.080853 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.080458 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ee2539c-afb2-4060-8461-61c4471b31d0" containerName="manager" Oct 30 18:43:51.083249 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.083228 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.123933 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.123889 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f67d6e91-06ca-4538-8c0e-40d5421544b6-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.123933 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.123941 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f67d6e91-06ca-4538-8c0e-40d5421544b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.124200 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.124032 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67d6e91-06ca-4538-8c0e-40d5421544b6-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.124200 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.124072 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f67d6e91-06ca-4538-8c0e-40d5421544b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.124200 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.124091 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f67d6e91-06ca-4538-8c0e-40d5421544b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.124200 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.124182 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67d6e91-06ca-4538-8c0e-40d5421544b6-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.124356 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.124205 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f67d6e91-06ca-4538-8c0e-40d5421544b6-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.124356 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.124227 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f67d6e91-06ca-4538-8c0e-40d5421544b6-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.124356 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.124293 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ffsh\" (UniqueName: \"kubernetes.io/projected/f67d6e91-06ca-4538-8c0e-40d5421544b6-kube-api-access-9ffsh\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.135394 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.135357 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 18:43:51.184809 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.184775 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-s874n\"" Oct 30 18:43:51.198267 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.198228 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd"] Oct 30 18:43:51.225010 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.224969 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f67d6e91-06ca-4538-8c0e-40d5421544b6-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.225243 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.225017 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f67d6e91-06ca-4538-8c0e-40d5421544b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.225243 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.225045 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67d6e91-06ca-4538-8c0e-40d5421544b6-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.225243 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.225067 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f67d6e91-06ca-4538-8c0e-40d5421544b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.225243 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.225089 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f67d6e91-06ca-4538-8c0e-40d5421544b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.225243 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.225116 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67d6e91-06ca-4538-8c0e-40d5421544b6-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.225243 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.225171 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f67d6e91-06ca-4538-8c0e-40d5421544b6-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.225243 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.225198 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f67d6e91-06ca-4538-8c0e-40d5421544b6-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.225629 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.225249 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ffsh\" (UniqueName: \"kubernetes.io/projected/f67d6e91-06ca-4538-8c0e-40d5421544b6-kube-api-access-9ffsh\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.225629 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.225517 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67d6e91-06ca-4538-8c0e-40d5421544b6-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.225732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.225654 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67d6e91-06ca-4538-8c0e-40d5421544b6-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.225815 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.225793 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f67d6e91-06ca-4538-8c0e-40d5421544b6-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.226075 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.225959 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f67d6e91-06ca-4538-8c0e-40d5421544b6-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.226158 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.226088 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f67d6e91-06ca-4538-8c0e-40d5421544b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.227887 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.227852 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f67d6e91-06ca-4538-8c0e-40d5421544b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.228201 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.228180 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f67d6e91-06ca-4538-8c0e-40d5421544b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.292604 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.292569 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f67d6e91-06ca-4538-8c0e-40d5421544b6-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.292760 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.292613 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ffsh\" (UniqueName: \"kubernetes.io/projected/f67d6e91-06ca-4538-8c0e-40d5421544b6-kube-api-access-9ffsh\") pod \"maas-default-gateway-openshift-default-66db998887-6khdd\" (UID: \"f67d6e91-06ca-4538-8c0e-40d5421544b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.396802 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.396700 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:51.823250 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:51.823193 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd"] Oct 30 18:43:51.825792 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:43:51.825755 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67d6e91_06ca_4538_8c0e_40d5421544b6.slice/crio-b6938641c74577c36301d87dd22385ffb9e2df430eab78b5fe22fd87b74eb67e WatchSource:0}: Error finding container b6938641c74577c36301d87dd22385ffb9e2df430eab78b5fe22fd87b74eb67e: Status 404 returned error can't find the container with id b6938641c74577c36301d87dd22385ffb9e2df430eab78b5fe22fd87b74eb67e Oct 30 18:43:52.549887 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:52.549849 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" event={"ID":"f67d6e91-06ca-4538-8c0e-40d5421544b6","Type":"ContainerStarted","Data":"b6938641c74577c36301d87dd22385ffb9e2df430eab78b5fe22fd87b74eb67e"} Oct 30 18:43:55.905399 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:55.905360 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:43:55.905812 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:55.905444 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:43:55.905812 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:55.905495 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:43:56.398176 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:43:56.398119 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:43:56.570218 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:56.570176 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" event={"ID":"f67d6e91-06ca-4538-8c0e-40d5421544b6","Type":"ContainerStarted","Data":"6dd9b3c63943d6303ee96b3dc97e8b861d86bdb2ad507918337c60eb6951002d"} Oct 30 18:43:56.750960 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:56.750895 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" podStartSLOduration=1.674441616 podStartE2EDuration="5.750871912s" totalImagesPullingTime="4.076430296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:56.748805643 +0000 UTC m=+596.775910601" watchObservedRunningTime="2026-10-30 18:43:56.750871912 +0000 UTC m=+596.777976871" Oct 30 18:43:57.397536 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:57.397498 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:43:57.398846 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:57.398815 4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-6khdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 30 18:43:57.398970 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:57.398870 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" podUID="f67d6e91-06ca-4538-8c0e-40d5421544b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 30 18:43:58.398152 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:58.398094 4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-6khdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 30 18:43:58.398569 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:58.398182 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" podUID="f67d6e91-06ca-4538-8c0e-40d5421544b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 30 18:43:59.398036 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:59.398004 4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-6khdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 30 18:43:59.398230 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:43:59.398061 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" podUID="f67d6e91-06ca-4538-8c0e-40d5421544b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 30 18:44:00.397611 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:00.397573 4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-6khdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 30 18:44:00.397801 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:00.397628 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" podUID="f67d6e91-06ca-4538-8c0e-40d5421544b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 30 18:44:01.397990 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:01.397950 4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-6khdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 30 18:44:01.398419 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:01.398003 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" podUID="f67d6e91-06ca-4538-8c0e-40d5421544b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 30 18:44:02.397929 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:02.397882 4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-6khdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 30 18:44:02.398122 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:02.397956 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" podUID="f67d6e91-06ca-4538-8c0e-40d5421544b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 30 18:44:03.397672 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:03.397633 4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-6khdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 30 18:44:03.397852 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:03.397696 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" podUID="f67d6e91-06ca-4538-8c0e-40d5421544b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 30 18:44:04.401820 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:04.401787 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:44:04.402243 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:04.402108 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:44:04.403018 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:04.402996 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-6khdd" Oct 30 18:44:06.205866 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.205829 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt"] Oct 30 18:44:06.209409 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.209385 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.318041 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.317998 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt"] Oct 30 18:44:06.392057 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.392017 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.392278 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.392065 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.392278 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.392085 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8256a12e-d979-4a98-8200-0b3937d26dae-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.392278 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.392100 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9mld\" (UniqueName: \"kubernetes.io/projected/8256a12e-d979-4a98-8200-0b3937d26dae-kube-api-access-h9mld\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.392278 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.392259 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.392433 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.392308 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.392433 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.392327 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.392433 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.392430 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8256a12e-d979-4a98-8200-0b3937d26dae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.392538 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.392456 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8256a12e-d979-4a98-8200-0b3937d26dae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.492893 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.492854 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.492893 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.492900 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.493221 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.492919 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8256a12e-d979-4a98-8200-0b3937d26dae-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.493221 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.492937 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9mld\" (UniqueName: \"kubernetes.io/projected/8256a12e-d979-4a98-8200-0b3937d26dae-kube-api-access-h9mld\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.493221 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.492969 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.493221 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.493102 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.493221 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.493178 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.493221 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.493220 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8256a12e-d979-4a98-8200-0b3937d26dae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.493540 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.493259 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8256a12e-d979-4a98-8200-0b3937d26dae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.493540 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.493409 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.493648 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.493548 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.493648 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.493587 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.493648 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.493603 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.494004 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.493983 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8256a12e-d979-4a98-8200-0b3937d26dae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.495535 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.495504 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.495838 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.495821 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8256a12e-d979-4a98-8200-0b3937d26dae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.554435 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.554397 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8256a12e-d979-4a98-8200-0b3937d26dae-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.554847 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.554824 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9mld\" (UniqueName: \"kubernetes.io/projected/8256a12e-d979-4a98-8200-0b3937d26dae-kube-api-access-h9mld\") pod \"maas-default-gateway-openshift-default-66db998887-nkqzt\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:06.823276 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:06.823171 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:07.254343 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:07.254318 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt"] Oct 30 18:44:07.257100 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:44:07.257072 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8256a12e_d979_4a98_8200_0b3937d26dae.slice/crio-ddcd7319168278436b71ebab5af221bb96795c69254c3fd45a732b35f349a4dc WatchSource:0}: Error finding container ddcd7319168278436b71ebab5af221bb96795c69254c3fd45a732b35f349a4dc: Status 404 returned error can't find the container with id ddcd7319168278436b71ebab5af221bb96795c69254c3fd45a732b35f349a4dc Oct 30 18:44:07.259265 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:07.259230 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:44:07.259345 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:07.259318 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:44:07.259395 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:07.259360 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:44:07.620493 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:07.620375 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" event={"ID":"8256a12e-d979-4a98-8200-0b3937d26dae","Type":"ContainerStarted","Data":"6d3f486feff032cf596c79d4491c6a85029f43cf3acffce51c5ad025dd650323"} Oct 30 18:44:07.620493 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:07.620436 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" event={"ID":"8256a12e-d979-4a98-8200-0b3937d26dae","Type":"ContainerStarted","Data":"ddcd7319168278436b71ebab5af221bb96795c69254c3fd45a732b35f349a4dc"} Oct 30 18:44:07.831159 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:07.828239 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:08.835081 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:08.835042 4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-nkqzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Oct 30 18:44:08.835721 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:08.835684 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" podUID="8256a12e-d979-4a98-8200-0b3937d26dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Oct 30 18:44:09.839161 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:09.828609 4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-nkqzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Oct 30 18:44:09.839161 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:09.828677 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" podUID="8256a12e-d979-4a98-8200-0b3937d26dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Oct 30 18:44:10.832631 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:10.832588 4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-nkqzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Oct 30 18:44:10.832936 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:10.832906 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" podUID="8256a12e-d979-4a98-8200-0b3937d26dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Oct 30 18:44:11.830161 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:11.824604 4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-nkqzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Oct 30 18:44:11.830161 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:11.824674 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" podUID="8256a12e-d979-4a98-8200-0b3937d26dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Oct 30 18:44:12.829666 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:12.829620 4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-nkqzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Oct 30 18:44:12.829982 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:12.829953 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" podUID="8256a12e-d979-4a98-8200-0b3937d26dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Oct 30 18:44:13.823875 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:13.823832 4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-nkqzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Oct 30 18:44:13.824313 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:13.823898 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" podUID="8256a12e-d979-4a98-8200-0b3937d26dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Oct 30 18:44:14.824553 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:14.824521 4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-nkqzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Oct 30 18:44:14.824931 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:14.824588 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" podUID="8256a12e-d979-4a98-8200-0b3937d26dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Oct 30 18:44:15.827647 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:15.827611 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:15.828121 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:15.827912 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:15.828753 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:15.828738 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 18:44:16.000653 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:16.000596 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" podStartSLOduration=10.000580426 podStartE2EDuration="10.000580426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:07.742635329 +0000 UTC m=+607.769740287" watchObservedRunningTime="2026-10-30 18:44:16.000580426 +0000 UTC m=+616.027685384" Oct 30 18:44:26.398240 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:44:26.398197 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:44:56.397906 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:44:56.397865 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:44:58.972123 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:58.972085 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp"] Oct 30 18:44:58.975463 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:58.975441 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp" Oct 30 18:44:59.027438 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:59.027412 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-620f2843\"" Oct 30 18:44:59.076951 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:59.076917 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-620f2843-dockercfg-dmbjv\"" Oct 30 18:44:59.077122 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:59.077008 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsz7b\" (UniqueName: \"kubernetes.io/projected/0a907a3e-3c6d-4a08-9a91-4d9f6dc12891-kube-api-access-rsz7b\") pod \"payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp\" (UID: \"0a907a3e-3c6d-4a08-9a91-4d9f6dc12891\") " pod="openshift-ingress/payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp" Oct 30 18:44:59.077122 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:59.077072 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a907a3e-3c6d-4a08-9a91-4d9f6dc12891-plugins-config-volume\") pod \"payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp\" (UID: \"0a907a3e-3c6d-4a08-9a91-4d9f6dc12891\") " pod="openshift-ingress/payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp" Oct 30 18:44:59.078351 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:59.078333 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp"] Oct 30 18:44:59.177954 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:59.177909 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsz7b\" (UniqueName: \"kubernetes.io/projected/0a907a3e-3c6d-4a08-9a91-4d9f6dc12891-kube-api-access-rsz7b\") pod \"payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp\" (UID: \"0a907a3e-3c6d-4a08-9a91-4d9f6dc12891\") " pod="openshift-ingress/payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp" Oct 30 18:44:59.178161 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:59.177999 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a907a3e-3c6d-4a08-9a91-4d9f6dc12891-plugins-config-volume\") pod \"payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp\" (UID: \"0a907a3e-3c6d-4a08-9a91-4d9f6dc12891\") " pod="openshift-ingress/payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp" Oct 30 18:44:59.178798 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:59.178778 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a907a3e-3c6d-4a08-9a91-4d9f6dc12891-plugins-config-volume\") pod \"payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp\" (UID: \"0a907a3e-3c6d-4a08-9a91-4d9f6dc12891\") " pod="openshift-ingress/payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp" Oct 30 18:44:59.239790 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:59.239712 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsz7b\" (UniqueName: \"kubernetes.io/projected/0a907a3e-3c6d-4a08-9a91-4d9f6dc12891-kube-api-access-rsz7b\") pod \"payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp\" (UID: \"0a907a3e-3c6d-4a08-9a91-4d9f6dc12891\") " pod="openshift-ingress/payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp" Oct 30 18:44:59.288029 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:59.287990 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp" Oct 30 18:44:59.663860 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:59.663834 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp"] Oct 30 18:44:59.666541 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:44:59.666505 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a907a3e_3c6d_4a08_9a91_4d9f6dc12891.slice/crio-82da28344299b3a6e50a5dad5d5cbd1ae373d07ed93ff81c4f2941e086cba1a9 WatchSource:0}: Error finding container 82da28344299b3a6e50a5dad5d5cbd1ae373d07ed93ff81c4f2941e086cba1a9: Status 404 returned error can't find the container with id 82da28344299b3a6e50a5dad5d5cbd1ae373d07ed93ff81c4f2941e086cba1a9 Oct 30 18:44:59.668506 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:59.668486 4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:44:59.868674 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:44:59.868642 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp" event={"ID":"0a907a3e-3c6d-4a08-9a91-4d9f6dc12891","Type":"ContainerStarted","Data":"82da28344299b3a6e50a5dad5d5cbd1ae373d07ed93ff81c4f2941e086cba1a9"} Oct 30 18:45:00.173890 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:00.173502 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2"] Oct 30 18:45:00.178301 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:00.178276 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc"] Oct 30 18:45:00.178415 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:00.178364 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2" Oct 30 18:45:00.181713 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:00.181688 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc" Oct 30 18:45:00.230455 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:00.230417 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:45:00.230624 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:00.230483 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:45:00.230793 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:00.230769 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:45:00.274068 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:00.274031 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2"] Oct 30 18:45:00.277057 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:00.276995 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc"] Oct 30 18:45:00.287947 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:00.287913 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa81fc3e-9916-4301-9f10-304927838d67-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc\" (UID: \"fa81fc3e-9916-4301-9f10-304927838d67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc" Oct 30 18:45:00.288173 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:00.287959 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2e78f4-4e72-4fec-9170-d4d5856ab281-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2\" (UID: \"5c2e78f4-4e72-4fec-9170-d4d5856ab281\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2" Oct 30 18:45:00.288173 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:00.288083 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74nk7\" (UniqueName: \"kubernetes.io/projected/5c2e78f4-4e72-4fec-9170-d4d5856ab281-kube-api-access-74nk7\") pod \"maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2\" (UID: \"5c2e78f4-4e72-4fec-9170-d4d5856ab281\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2" Oct 30 18:45:00.288270 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:00.288216 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzx74\" (UniqueName: \"kubernetes.io/projected/fa81fc3e-9916-4301-9f10-304927838d67-kube-api-access-lzx74\") pod \"maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc\" (UID: \"fa81fc3e-9916-4301-9f10-304927838d67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc" Oct 30 18:45:00.389462 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:00.389430 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74nk7\" (UniqueName: \"kubernetes.io/projected/5c2e78f4-4e72-4fec-9170-d4d5856ab281-kube-api-access-74nk7\") pod \"maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2\" (UID: \"5c2e78f4-4e72-4fec-9170-d4d5856ab281\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2" Oct 30 18:45:00.389674 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:00.389519 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzx74\" (UniqueName: \"kubernetes.io/projected/fa81fc3e-9916-4301-9f10-304927838d67-kube-api-access-lzx74\") pod \"maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc\" (UID: \"fa81fc3e-9916-4301-9f10-304927838d67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc" Oct 30 18:45:00.389674 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:00.389572 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa81fc3e-9916-4301-9f10-304927838d67-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc\" (UID: \"fa81fc3e-9916-4301-9f10-304927838d67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc" Oct 30 18:45:00.389674 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:00.389603 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2e78f4-4e72-4fec-9170-d4d5856ab281-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2\" (UID: \"5c2e78f4-4e72-4fec-9170-d4d5856ab281\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2" Oct 30 18:45:00.390277 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:00.390253 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2e78f4-4e72-4fec-9170-d4d5856ab281-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2\" (UID: \"5c2e78f4-4e72-4fec-9170-d4d5856ab281\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2" Oct 30 18:45:00.390425 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:00.390407 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa81fc3e-9916-4301-9f10-304927838d67-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc\" (UID: \"fa81fc3e-9916-4301-9f10-304927838d67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc" Oct 30 18:45:00.452031 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:00.451956 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzx74\" (UniqueName: \"kubernetes.io/projected/fa81fc3e-9916-4301-9f10-304927838d67-kube-api-access-lzx74\") pod \"maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc\" (UID: \"fa81fc3e-9916-4301-9f10-304927838d67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc" Oct 30 18:45:00.457506 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:00.457478 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74nk7\" (UniqueName: \"kubernetes.io/projected/5c2e78f4-4e72-4fec-9170-d4d5856ab281-kube-api-access-74nk7\") pod \"maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2\" (UID: \"5c2e78f4-4e72-4fec-9170-d4d5856ab281\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2" Oct 30 18:45:00.493541 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:00.493514 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2" Oct 30 18:45:00.500422 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:00.500397 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc" Oct 30 18:45:00.865953 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:00.865923 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2"] Oct 30 18:45:00.869328 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:45:00.869298 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2e78f4_4e72_4fec_9170_d4d5856ab281.slice/crio-c7adf2e7f0bde18a35f77be63661c4ed46d9455a604e8b70d28b3cd0a4b087e8 WatchSource:0}: Error finding container c7adf2e7f0bde18a35f77be63661c4ed46d9455a604e8b70d28b3cd0a4b087e8: Status 404 returned error can't find the container with id c7adf2e7f0bde18a35f77be63661c4ed46d9455a604e8b70d28b3cd0a4b087e8 Oct 30 18:45:00.916996 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:00.916954 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc"] Oct 30 18:45:00.921302 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:45:00.921262 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa81fc3e_9916_4301_9f10_304927838d67.slice/crio-ed7667c1d1b6a10401ca65b58d018546050f59ff2f7268f1ca94aa66a2855f37 WatchSource:0}: Error finding container ed7667c1d1b6a10401ca65b58d018546050f59ff2f7268f1ca94aa66a2855f37: Status 404 returned error can't find the container with id ed7667c1d1b6a10401ca65b58d018546050f59ff2f7268f1ca94aa66a2855f37 Oct 30 18:45:01.879705 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:01.879662 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc" event={"ID":"fa81fc3e-9916-4301-9f10-304927838d67","Type":"ContainerStarted","Data":"ed7667c1d1b6a10401ca65b58d018546050f59ff2f7268f1ca94aa66a2855f37"} Oct 30 18:45:01.881075 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:01.881023 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2" event={"ID":"5c2e78f4-4e72-4fec-9170-d4d5856ab281","Type":"ContainerStarted","Data":"c7adf2e7f0bde18a35f77be63661c4ed46d9455a604e8b70d28b3cd0a4b087e8"} Oct 30 18:45:02.887452 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:02.887347 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp" event={"ID":"0a907a3e-3c6d-4a08-9a91-4d9f6dc12891","Type":"ContainerStarted","Data":"9a31de07d5f3838de324454e7f119f5ddcb055b261fe61140dc457373713fc40"} Oct 30 18:45:02.887896 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:02.887500 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp" Oct 30 18:45:03.004224 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:03.004162 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp" podStartSLOduration=2.037913124 podStartE2EDuration="5.004118893s" totalImagesPullingTime="2.966205769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:03.003170271 +0000 UTC m=+663.030275236" watchObservedRunningTime="2026-10-30 18:45:03.004118893 +0000 UTC m=+663.031223851" Oct 30 18:45:05.903460 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:05.903421 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc" event={"ID":"fa81fc3e-9916-4301-9f10-304927838d67","Type":"ContainerStarted","Data":"515f39d69cb9472d97ca1cccf926e7623ada710f12122da113e6f463663cbe60"} Oct 30 18:45:05.905080 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:05.905047 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2" event={"ID":"5c2e78f4-4e72-4fec-9170-d4d5856ab281","Type":"ContainerStarted","Data":"189d3525c4bbdaa9def7a7bde2297e434eac99d03c6487c95a1e227d685f5033"} Oct 30 18:45:06.020765 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:06.020702 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc" podStartSLOduration=1.700917158 podStartE2EDuration="6.020686207s" totalImagesPullingTime="4.319769049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:06.019069857 +0000 UTC m=+666.046174817" watchObservedRunningTime="2026-10-30 18:45:06.020686207 +0000 UTC m=+666.047791167" Oct 30 18:45:06.135487 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:06.135427 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2" podStartSLOduration=1.8540047610000001 podStartE2EDuration="6.135408761s" totalImagesPullingTime="4.281404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:06.133973424 +0000 UTC m=+666.161078383" watchObservedRunningTime="2026-10-30 18:45:06.135408761 +0000 UTC m=+666.162513721" Oct 30 18:45:06.910859 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:06.910822 4531 generic.go:356] "Generic (PLEG): container finished" podID="fa81fc3e-9916-4301-9f10-304927838d67" containerID="515f39d69cb9472d97ca1cccf926e7623ada710f12122da113e6f463663cbe60" exitCode=60 Oct 30 18:45:06.911354 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:06.910910 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc" event={"ID":"fa81fc3e-9916-4301-9f10-304927838d67","Type":"ContainerDied","Data":"515f39d69cb9472d97ca1cccf926e7623ada710f12122da113e6f463663cbe60"} Oct 30 18:45:06.911354 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:06.911111 4531 scope.go:122] "RemoveContainer" containerID="515f39d69cb9472d97ca1cccf926e7623ada710f12122da113e6f463663cbe60" Oct 30 18:45:06.912664 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:06.912639 4531 generic.go:356] "Generic (PLEG): container finished" podID="5c2e78f4-4e72-4fec-9170-d4d5856ab281" containerID="189d3525c4bbdaa9def7a7bde2297e434eac99d03c6487c95a1e227d685f5033" exitCode=60 Oct 30 18:45:06.912759 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:06.912724 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2" event={"ID":"5c2e78f4-4e72-4fec-9170-d4d5856ab281","Type":"ContainerDied","Data":"189d3525c4bbdaa9def7a7bde2297e434eac99d03c6487c95a1e227d685f5033"} Oct 30 18:45:06.912935 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:06.912918 4531 scope.go:122] "RemoveContainer" containerID="189d3525c4bbdaa9def7a7bde2297e434eac99d03c6487c95a1e227d685f5033" Oct 30 18:45:07.925825 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:07.925785 4531 generic.go:356] "Generic (PLEG): container finished" podID="fa81fc3e-9916-4301-9f10-304927838d67" containerID="4a6dd242e03b92e703b65f1c95dc7b625f6d265c9408ec35cc4177b5ed0d7118" exitCode=60 Oct 30 18:45:07.926355 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:07.925864 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc" event={"ID":"fa81fc3e-9916-4301-9f10-304927838d67","Type":"ContainerDied","Data":"4a6dd242e03b92e703b65f1c95dc7b625f6d265c9408ec35cc4177b5ed0d7118"} Oct 30 18:45:07.926355 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:07.925979 4531 scope.go:122] "RemoveContainer" containerID="515f39d69cb9472d97ca1cccf926e7623ada710f12122da113e6f463663cbe60" Oct 30 18:45:07.926355 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:07.926245 4531 scope.go:122] "RemoveContainer" containerID="4a6dd242e03b92e703b65f1c95dc7b625f6d265c9408ec35cc4177b5ed0d7118" Oct 30 18:45:07.926513 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:45:07.926497 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc_odh-ai-gateway-infra(fa81fc3e-9916-4301-9f10-304927838d67)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc" podUID="fa81fc3e-9916-4301-9f10-304927838d67" Oct 30 18:45:07.927886 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:07.927866 4531 generic.go:356] "Generic (PLEG): container finished" podID="5c2e78f4-4e72-4fec-9170-d4d5856ab281" containerID="db54300d5284fbea967b362e5b1cba78e58ee1d68128773380bc899f6717823b" exitCode=60 Oct 30 18:45:07.927993 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:07.927934 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2" event={"ID":"5c2e78f4-4e72-4fec-9170-d4d5856ab281","Type":"ContainerDied","Data":"db54300d5284fbea967b362e5b1cba78e58ee1d68128773380bc899f6717823b"} Oct 30 18:45:07.928269 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:07.928256 4531 scope.go:122] "RemoveContainer" containerID="db54300d5284fbea967b362e5b1cba78e58ee1d68128773380bc899f6717823b" Oct 30 18:45:07.928485 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:45:07.928468 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2_odh-ai-gateway-infra(5c2e78f4-4e72-4fec-9170-d4d5856ab281)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2" podUID="5c2e78f4-4e72-4fec-9170-d4d5856ab281" Oct 30 18:45:07.968576 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:07.968554 4531 scope.go:122] "RemoveContainer" containerID="189d3525c4bbdaa9def7a7bde2297e434eac99d03c6487c95a1e227d685f5033" Oct 30 18:45:08.934773 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:08.934734 4531 scope.go:122] "RemoveContainer" containerID="4a6dd242e03b92e703b65f1c95dc7b625f6d265c9408ec35cc4177b5ed0d7118" Oct 30 18:45:08.935279 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:45:08.934994 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc_odh-ai-gateway-infra(fa81fc3e-9916-4301-9f10-304927838d67)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc" podUID="fa81fc3e-9916-4301-9f10-304927838d67" Oct 30 18:45:08.936499 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:08.936474 4531 scope.go:122] "RemoveContainer" containerID="db54300d5284fbea967b362e5b1cba78e58ee1d68128773380bc899f6717823b" Oct 30 18:45:08.936747 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:45:08.936725 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2_odh-ai-gateway-infra(5c2e78f4-4e72-4fec-9170-d4d5856ab281)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2" podUID="5c2e78f4-4e72-4fec-9170-d4d5856ab281" Oct 30 18:45:13.896372 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:13.896288 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp" Oct 30 18:45:19.092597 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:19.092562 4531 scope.go:122] "RemoveContainer" containerID="db54300d5284fbea967b362e5b1cba78e58ee1d68128773380bc899f6717823b" Oct 30 18:45:19.986094 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:19.986055 4531 generic.go:356] "Generic (PLEG): container finished" podID="5c2e78f4-4e72-4fec-9170-d4d5856ab281" containerID="5ab46a95e6da1fee4c768761d1ea38cdfb54f1a3f1f46d71fda41052b9442652" exitCode=60 Oct 30 18:45:19.986290 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:19.986106 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2" event={"ID":"5c2e78f4-4e72-4fec-9170-d4d5856ab281","Type":"ContainerDied","Data":"5ab46a95e6da1fee4c768761d1ea38cdfb54f1a3f1f46d71fda41052b9442652"} Oct 30 18:45:19.986290 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:19.986188 4531 scope.go:122] "RemoveContainer" containerID="db54300d5284fbea967b362e5b1cba78e58ee1d68128773380bc899f6717823b" Oct 30 18:45:19.986550 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:19.986531 4531 scope.go:122] "RemoveContainer" containerID="5ab46a95e6da1fee4c768761d1ea38cdfb54f1a3f1f46d71fda41052b9442652" Oct 30 18:45:19.986818 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:45:19.986793 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2_odh-ai-gateway-infra(5c2e78f4-4e72-4fec-9170-d4d5856ab281)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2" podUID="5c2e78f4-4e72-4fec-9170-d4d5856ab281" Oct 30 18:45:21.092586 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:21.092542 4531 scope.go:122] "RemoveContainer" containerID="4a6dd242e03b92e703b65f1c95dc7b625f6d265c9408ec35cc4177b5ed0d7118" Oct 30 18:45:21.114745 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:21.114709 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2"] Oct 30 18:45:21.265305 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:21.265271 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2" Oct 30 18:45:21.380596 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:21.380479 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c2e78f4-4e72-4fec-9170-d4d5856ab281-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2e78f4-4e72-4fec-9170-d4d5856ab281-maas-api-service-ca\") pod \"5c2e78f4-4e72-4fec-9170-d4d5856ab281\" (UID: \"5c2e78f4-4e72-4fec-9170-d4d5856ab281\") " Oct 30 18:45:21.380596 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:21.380565 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c2e78f4-4e72-4fec-9170-d4d5856ab281-kube-api-access-74nk7\" (UniqueName: \"kubernetes.io/projected/5c2e78f4-4e72-4fec-9170-d4d5856ab281-kube-api-access-74nk7\") pod \"5c2e78f4-4e72-4fec-9170-d4d5856ab281\" (UID: \"5c2e78f4-4e72-4fec-9170-d4d5856ab281\") " Oct 30 18:45:21.381031 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:21.381004 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2e78f4-4e72-4fec-9170-d4d5856ab281-maas-api-service-ca" pod "5c2e78f4-4e72-4fec-9170-d4d5856ab281" (UID: "5c2e78f4-4e72-4fec-9170-d4d5856ab281"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:21.384296 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:21.384253 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2e78f4-4e72-4fec-9170-d4d5856ab281-kube-api-access-74nk7" pod "5c2e78f4-4e72-4fec-9170-d4d5856ab281" (UID: "5c2e78f4-4e72-4fec-9170-d4d5856ab281"). InnerVolumeSpecName "kube-api-access-74nk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:21.481294 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:21.481264 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74nk7\" (UniqueName: \"kubernetes.io/projected/5c2e78f4-4e72-4fec-9170-d4d5856ab281-kube-api-access-74nk7\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:45:21.481294 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:21.481292 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2e78f4-4e72-4fec-9170-d4d5856ab281-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:45:21.999644 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:21.999611 4531 generic.go:356] "Generic (PLEG): container finished" podID="fa81fc3e-9916-4301-9f10-304927838d67" containerID="a7fc30fb76bd4cfda5850585182ee96d88791b0786b15db2e851bdb57aad5aed" exitCode=60 Oct 30 18:45:21.999840 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:21.999686 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc" event={"ID":"fa81fc3e-9916-4301-9f10-304927838d67","Type":"ContainerDied","Data":"a7fc30fb76bd4cfda5850585182ee96d88791b0786b15db2e851bdb57aad5aed"} Oct 30 18:45:21.999840 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:21.999770 4531 scope.go:122] "RemoveContainer" containerID="4a6dd242e03b92e703b65f1c95dc7b625f6d265c9408ec35cc4177b5ed0d7118" Oct 30 18:45:22.000156 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:22.000104 4531 scope.go:122] "RemoveContainer" containerID="a7fc30fb76bd4cfda5850585182ee96d88791b0786b15db2e851bdb57aad5aed" Oct 30 18:45:22.000419 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:45:22.000397 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc_odh-ai-gateway-infra(fa81fc3e-9916-4301-9f10-304927838d67)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc" podUID="fa81fc3e-9916-4301-9f10-304927838d67" Oct 30 18:45:22.001359 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:22.001326 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2" event={"ID":"5c2e78f4-4e72-4fec-9170-d4d5856ab281","Type":"ContainerDied","Data":"c7adf2e7f0bde18a35f77be63661c4ed46d9455a604e8b70d28b3cd0a4b087e8"} Oct 30 18:45:22.001432 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:22.001383 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2" Oct 30 18:45:22.018656 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:22.018624 4531 scope.go:122] "RemoveContainer" containerID="5ab46a95e6da1fee4c768761d1ea38cdfb54f1a3f1f46d71fda41052b9442652" Oct 30 18:45:22.225276 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:22.225186 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2"] Oct 30 18:45:22.232789 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:22.232756 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-def78c01-29889765-xcgg2"] Oct 30 18:45:23.126861 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:23.126827 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc"] Oct 30 18:45:23.271153 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:23.271098 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc" Oct 30 18:45:23.298884 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:23.298849 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa81fc3e-9916-4301-9f10-304927838d67-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa81fc3e-9916-4301-9f10-304927838d67-maas-api-service-ca\") pod \"fa81fc3e-9916-4301-9f10-304927838d67\" (UID: \"fa81fc3e-9916-4301-9f10-304927838d67\") " Oct 30 18:45:23.299044 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:23.298931 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa81fc3e-9916-4301-9f10-304927838d67-kube-api-access-lzx74\" (UniqueName: \"kubernetes.io/projected/fa81fc3e-9916-4301-9f10-304927838d67-kube-api-access-lzx74\") pod \"fa81fc3e-9916-4301-9f10-304927838d67\" (UID: \"fa81fc3e-9916-4301-9f10-304927838d67\") " Oct 30 18:45:23.299395 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:23.299367 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa81fc3e-9916-4301-9f10-304927838d67-maas-api-service-ca" pod "fa81fc3e-9916-4301-9f10-304927838d67" (UID: "fa81fc3e-9916-4301-9f10-304927838d67"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:23.301759 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:23.301726 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa81fc3e-9916-4301-9f10-304927838d67-kube-api-access-lzx74" pod "fa81fc3e-9916-4301-9f10-304927838d67" (UID: "fa81fc3e-9916-4301-9f10-304927838d67"). InnerVolumeSpecName "kube-api-access-lzx74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:23.400071 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:23.399972 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzx74\" (UniqueName: \"kubernetes.io/projected/fa81fc3e-9916-4301-9f10-304927838d67-kube-api-access-lzx74\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:45:23.400071 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:23.400012 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa81fc3e-9916-4301-9f10-304927838d67-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:45:24.013479 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:24.013447 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc" Oct 30 18:45:24.013479 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:24.013476 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc" event={"ID":"fa81fc3e-9916-4301-9f10-304927838d67","Type":"ContainerDied","Data":"ed7667c1d1b6a10401ca65b58d018546050f59ff2f7268f1ca94aa66a2855f37"} Oct 30 18:45:24.013737 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:24.013688 4531 scope.go:122] "RemoveContainer" containerID="a7fc30fb76bd4cfda5850585182ee96d88791b0786b15db2e851bdb57aad5aed" Oct 30 18:45:24.097790 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:24.097745 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c2e78f4-4e72-4fec-9170-d4d5856ab281" path="/var/lib/kubelet/pods/5c2e78f4-4e72-4fec-9170-d4d5856ab281/volumes" Oct 30 18:45:24.176574 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:24.176541 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc"] Oct 30 18:45:24.225995 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:24.225956 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-16a49490-29889765-sbpjc"] Oct 30 18:45:24.932936 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:24.932903 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a49490-v2htg"] Oct 30 18:45:24.933448 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:24.933295 4531 cpu_manager.go:414] "containerMap: removing container" podUID="5c2e78f4-4e72-4fec-9170-d4d5856ab281" containerName="cleanup" Oct 30 18:45:24.933448 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:24.933308 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2e78f4-4e72-4fec-9170-d4d5856ab281" containerName="cleanup" Oct 30 18:45:24.933448 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:24.933320 4531 cpu_manager.go:414] "containerMap: removing container" podUID="fa81fc3e-9916-4301-9f10-304927838d67" containerName="cleanup" Oct 30 18:45:24.933448 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:24.933325 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa81fc3e-9916-4301-9f10-304927838d67" containerName="cleanup" Oct 30 18:45:24.933448 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:24.933336 4531 cpu_manager.go:414] "containerMap: removing container" podUID="5c2e78f4-4e72-4fec-9170-d4d5856ab281" containerName="cleanup" Oct 30 18:45:24.933448 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:24.933341 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2e78f4-4e72-4fec-9170-d4d5856ab281" containerName="cleanup" Oct 30 18:45:24.933448 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:24.933374 4531 cpu_manager.go:414] "containerMap: removing container" podUID="fa81fc3e-9916-4301-9f10-304927838d67" containerName="cleanup" Oct 30 18:45:24.933448 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:24.933379 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa81fc3e-9916-4301-9f10-304927838d67" containerName="cleanup" Oct 30 18:45:24.933448 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:24.933425 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2e78f4-4e72-4fec-9170-d4d5856ab281" containerName="cleanup" Oct 30 18:45:24.933448 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:24.933436 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2e78f4-4e72-4fec-9170-d4d5856ab281" containerName="cleanup" Oct 30 18:45:24.933448 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:24.933442 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa81fc3e-9916-4301-9f10-304927838d67" containerName="cleanup" Oct 30 18:45:24.933448 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:24.933448 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa81fc3e-9916-4301-9f10-304927838d67" containerName="cleanup" Oct 30 18:45:24.937655 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:24.937634 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a49490-v2htg" Oct 30 18:45:24.989876 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:24.989845 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:45:25.013573 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:25.013523 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pjnt\" (UniqueName: \"kubernetes.io/projected/2f4a27a8-bfc4-440a-996f-e0bce34bdc82-kube-api-access-8pjnt\") pod \"maas-api-revoke-keys-e2e-migrate-16a49490-v2htg\" (UID: \"2f4a27a8-bfc4-440a-996f-e0bce34bdc82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a49490-v2htg" Oct 30 18:45:25.013776 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:25.013590 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f4a27a8-bfc4-440a-996f-e0bce34bdc82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-16a49490-v2htg\" (UID: \"2f4a27a8-bfc4-440a-996f-e0bce34bdc82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a49490-v2htg" Oct 30 18:45:25.013776 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:25.013759 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f4a27a8-bfc4-440a-996f-e0bce34bdc82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-16a49490-v2htg\" (UID: \"2f4a27a8-bfc4-440a-996f-e0bce34bdc82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a49490-v2htg" Oct 30 18:45:25.039494 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:25.039453 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:45:25.039494 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:25.039483 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:45:25.039494 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:25.039458 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:45:25.041099 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:25.041073 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a49490-v2htg"] Oct 30 18:45:25.115218 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:25.115171 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f4a27a8-bfc4-440a-996f-e0bce34bdc82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-16a49490-v2htg\" (UID: \"2f4a27a8-bfc4-440a-996f-e0bce34bdc82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a49490-v2htg" Oct 30 18:45:25.115403 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:25.115242 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pjnt\" (UniqueName: \"kubernetes.io/projected/2f4a27a8-bfc4-440a-996f-e0bce34bdc82-kube-api-access-8pjnt\") pod \"maas-api-revoke-keys-e2e-migrate-16a49490-v2htg\" (UID: \"2f4a27a8-bfc4-440a-996f-e0bce34bdc82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a49490-v2htg" Oct 30 18:45:25.115403 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:25.115277 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f4a27a8-bfc4-440a-996f-e0bce34bdc82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-16a49490-v2htg\" (UID: \"2f4a27a8-bfc4-440a-996f-e0bce34bdc82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a49490-v2htg" Oct 30 18:45:25.116013 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:25.115992 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f4a27a8-bfc4-440a-996f-e0bce34bdc82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-16a49490-v2htg\" (UID: \"2f4a27a8-bfc4-440a-996f-e0bce34bdc82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a49490-v2htg" Oct 30 18:45:25.117769 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:25.117746 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f4a27a8-bfc4-440a-996f-e0bce34bdc82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-16a49490-v2htg\" (UID: \"2f4a27a8-bfc4-440a-996f-e0bce34bdc82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a49490-v2htg" Oct 30 18:45:25.177091 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:25.177063 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pjnt\" (UniqueName: \"kubernetes.io/projected/2f4a27a8-bfc4-440a-996f-e0bce34bdc82-kube-api-access-8pjnt\") pod \"maas-api-revoke-keys-e2e-migrate-16a49490-v2htg\" (UID: \"2f4a27a8-bfc4-440a-996f-e0bce34bdc82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a49490-v2htg" Oct 30 18:45:25.250065 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:25.250033 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a49490-v2htg" Oct 30 18:45:25.627502 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:25.627403 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a49490-v2htg"] Oct 30 18:45:25.630563 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:45:25.630529 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4a27a8_bfc4_440a_996f_e0bce34bdc82.slice/crio-7a6e09e16651536b0bb4db02ce84be18ad0f4967552dc29dfe36d19ec84faeca WatchSource:0}: Error finding container 7a6e09e16651536b0bb4db02ce84be18ad0f4967552dc29dfe36d19ec84faeca: Status 404 returned error can't find the container with id 7a6e09e16651536b0bb4db02ce84be18ad0f4967552dc29dfe36d19ec84faeca Oct 30 18:45:26.031234 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:26.031193 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a49490-v2htg" event={"ID":"2f4a27a8-bfc4-440a-996f-e0bce34bdc82","Type":"ContainerStarted","Data":"36a467da2c7acf74ade90624cbdfcaa857c6e3fe12397ac329dfed87383b5433"} Oct 30 18:45:26.031656 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:26.031279 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a49490-v2htg" event={"ID":"2f4a27a8-bfc4-440a-996f-e0bce34bdc82","Type":"ContainerStarted","Data":"7a6e09e16651536b0bb4db02ce84be18ad0f4967552dc29dfe36d19ec84faeca"} Oct 30 18:45:26.096653 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:26.096617 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa81fc3e-9916-4301-9f10-304927838d67" path="/var/lib/kubelet/pods/fa81fc3e-9916-4301-9f10-304927838d67/volumes" Oct 30 18:45:26.153838 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:26.153775 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a49490-v2htg" podStartSLOduration=2.153757471 podStartE2EDuration="2.153757471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:26.152528707 +0000 UTC m=+686.179633667" watchObservedRunningTime="2026-10-30 18:45:26.153757471 +0000 UTC m=+686.180862430" Oct 30 18:45:26.399812 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:45:26.399779 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:45:27.037713 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:27.037671 4531 generic.go:356] "Generic (PLEG): container finished" podID="2f4a27a8-bfc4-440a-996f-e0bce34bdc82" containerID="36a467da2c7acf74ade90624cbdfcaa857c6e3fe12397ac329dfed87383b5433" exitCode=0 Oct 30 18:45:27.038155 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:27.037768 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a49490-v2htg" event={"ID":"2f4a27a8-bfc4-440a-996f-e0bce34bdc82","Type":"ContainerDied","Data":"36a467da2c7acf74ade90624cbdfcaa857c6e3fe12397ac329dfed87383b5433"} Oct 30 18:45:28.184495 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:28.184460 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a49490-v2htg" Oct 30 18:45:28.250161 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:28.250096 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f4a27a8-bfc4-440a-996f-e0bce34bdc82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f4a27a8-bfc4-440a-996f-e0bce34bdc82-maas-api-openshift-ca\") pod \"2f4a27a8-bfc4-440a-996f-e0bce34bdc82\" (UID: \"2f4a27a8-bfc4-440a-996f-e0bce34bdc82\") " Oct 30 18:45:28.250307 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:28.250197 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f4a27a8-bfc4-440a-996f-e0bce34bdc82-kube-api-access-8pjnt\" (UniqueName: \"kubernetes.io/projected/2f4a27a8-bfc4-440a-996f-e0bce34bdc82-kube-api-access-8pjnt\") pod \"2f4a27a8-bfc4-440a-996f-e0bce34bdc82\" (UID: \"2f4a27a8-bfc4-440a-996f-e0bce34bdc82\") " Oct 30 18:45:28.250307 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:28.250267 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f4a27a8-bfc4-440a-996f-e0bce34bdc82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f4a27a8-bfc4-440a-996f-e0bce34bdc82-maas-api-xks-ca\") pod \"2f4a27a8-bfc4-440a-996f-e0bce34bdc82\" (UID: \"2f4a27a8-bfc4-440a-996f-e0bce34bdc82\") " Oct 30 18:45:28.250525 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:28.250503 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f4a27a8-bfc4-440a-996f-e0bce34bdc82-maas-api-openshift-ca" pod "2f4a27a8-bfc4-440a-996f-e0bce34bdc82" (UID: "2f4a27a8-bfc4-440a-996f-e0bce34bdc82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:28.252652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:28.252614 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f4a27a8-bfc4-440a-996f-e0bce34bdc82-maas-api-xks-ca" pod "2f4a27a8-bfc4-440a-996f-e0bce34bdc82" (UID: "2f4a27a8-bfc4-440a-996f-e0bce34bdc82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:45:28.252652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:28.252648 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4a27a8-bfc4-440a-996f-e0bce34bdc82-kube-api-access-8pjnt" pod "2f4a27a8-bfc4-440a-996f-e0bce34bdc82" (UID: "2f4a27a8-bfc4-440a-996f-e0bce34bdc82"). InnerVolumeSpecName "kube-api-access-8pjnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:28.351721 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:28.351625 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pjnt\" (UniqueName: \"kubernetes.io/projected/2f4a27a8-bfc4-440a-996f-e0bce34bdc82-kube-api-access-8pjnt\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:45:28.351721 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:28.351662 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f4a27a8-bfc4-440a-996f-e0bce34bdc82-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:45:28.351721 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:28.351672 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f4a27a8-bfc4-440a-996f-e0bce34bdc82-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:45:29.049950 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:29.049913 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a49490-v2htg" Oct 30 18:45:29.050161 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:29.049950 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a49490-v2htg" event={"ID":"2f4a27a8-bfc4-440a-996f-e0bce34bdc82","Type":"ContainerDied","Data":"7a6e09e16651536b0bb4db02ce84be18ad0f4967552dc29dfe36d19ec84faeca"} Oct 30 18:45:29.050161 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:29.049999 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a6e09e16651536b0bb4db02ce84be18ad0f4967552dc29dfe36d19ec84faeca" Oct 30 18:45:34.503555 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:34.503512 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj"] Oct 30 18:45:34.503967 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:34.503856 4531 cpu_manager.go:414] "containerMap: removing container" podUID="fa81fc3e-9916-4301-9f10-304927838d67" containerName="cleanup" Oct 30 18:45:34.503967 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:34.503867 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa81fc3e-9916-4301-9f10-304927838d67" containerName="cleanup" Oct 30 18:45:34.503967 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:34.503880 4531 cpu_manager.go:414] "containerMap: removing container" podUID="5c2e78f4-4e72-4fec-9170-d4d5856ab281" containerName="cleanup" Oct 30 18:45:34.503967 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:34.503885 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2e78f4-4e72-4fec-9170-d4d5856ab281" containerName="cleanup" Oct 30 18:45:34.503967 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:34.503892 4531 cpu_manager.go:414] "containerMap: removing container" podUID="2f4a27a8-bfc4-440a-996f-e0bce34bdc82" containerName="revoke-keys" Oct 30 18:45:34.503967 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:34.503897 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f4a27a8-bfc4-440a-996f-e0bce34bdc82" containerName="revoke-keys" Oct 30 18:45:34.503967 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:34.503969 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f4a27a8-bfc4-440a-996f-e0bce34bdc82" containerName="revoke-keys" Oct 30 18:45:34.504211 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:34.503981 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2e78f4-4e72-4fec-9170-d4d5856ab281" containerName="cleanup" Oct 30 18:45:34.504211 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:34.503988 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa81fc3e-9916-4301-9f10-304927838d67" containerName="cleanup" Oct 30 18:45:34.511802 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:34.511766 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj" Oct 30 18:45:34.611144 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:34.611093 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dacd0248-6d7e-4835-99dd-7aaa5ce74477-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj\" (UID: \"dacd0248-6d7e-4835-99dd-7aaa5ce74477\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj" Oct 30 18:45:34.611144 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:34.611141 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmbgh\" (UniqueName: \"kubernetes.io/projected/dacd0248-6d7e-4835-99dd-7aaa5ce74477-kube-api-access-mmbgh\") pod \"maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj\" (UID: \"dacd0248-6d7e-4835-99dd-7aaa5ce74477\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj" Oct 30 18:45:34.611334 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:34.611171 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dacd0248-6d7e-4835-99dd-7aaa5ce74477-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj\" (UID: \"dacd0248-6d7e-4835-99dd-7aaa5ce74477\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj" Oct 30 18:45:34.618063 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:34.618029 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:45:34.618240 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:34.618084 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:45:34.618240 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:34.618124 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:45:34.618240 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:34.618193 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:45:34.619536 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:34.619517 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj"] Oct 30 18:45:34.711979 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:34.711940 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dacd0248-6d7e-4835-99dd-7aaa5ce74477-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj\" (UID: \"dacd0248-6d7e-4835-99dd-7aaa5ce74477\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj" Oct 30 18:45:34.711979 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:34.711979 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmbgh\" (UniqueName: \"kubernetes.io/projected/dacd0248-6d7e-4835-99dd-7aaa5ce74477-kube-api-access-mmbgh\") pod \"maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj\" (UID: \"dacd0248-6d7e-4835-99dd-7aaa5ce74477\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj" Oct 30 18:45:34.712282 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:34.712020 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dacd0248-6d7e-4835-99dd-7aaa5ce74477-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj\" (UID: \"dacd0248-6d7e-4835-99dd-7aaa5ce74477\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj" Oct 30 18:45:34.712720 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:34.712695 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dacd0248-6d7e-4835-99dd-7aaa5ce74477-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj\" (UID: \"dacd0248-6d7e-4835-99dd-7aaa5ce74477\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj" Oct 30 18:45:34.714504 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:34.714486 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dacd0248-6d7e-4835-99dd-7aaa5ce74477-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj\" (UID: \"dacd0248-6d7e-4835-99dd-7aaa5ce74477\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj" Oct 30 18:45:34.773275 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:34.773204 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmbgh\" (UniqueName: \"kubernetes.io/projected/dacd0248-6d7e-4835-99dd-7aaa5ce74477-kube-api-access-mmbgh\") pod \"maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj\" (UID: \"dacd0248-6d7e-4835-99dd-7aaa5ce74477\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj" Oct 30 18:45:34.824744 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:34.824713 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj" Oct 30 18:45:35.200139 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:35.200032 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj"] Oct 30 18:45:35.203576 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:45:35.203545 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddacd0248_6d7e_4835_99dd_7aaa5ce74477.slice/crio-423664f6d0c067e86ff3a88d43496ab7e7e585dc2afac593cc472526292328fb WatchSource:0}: Error finding container 423664f6d0c067e86ff3a88d43496ab7e7e585dc2afac593cc472526292328fb: Status 404 returned error can't find the container with id 423664f6d0c067e86ff3a88d43496ab7e7e585dc2afac593cc472526292328fb Oct 30 18:45:36.085306 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:36.085276 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj" event={"ID":"dacd0248-6d7e-4835-99dd-7aaa5ce74477","Type":"ContainerStarted","Data":"adcc63ff33ca6bcc86bc0c949842e0a95c02ed885bb4de346bd92b7da525a368"} Oct 30 18:45:36.085642 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:36.085325 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj" event={"ID":"dacd0248-6d7e-4835-99dd-7aaa5ce74477","Type":"ContainerStarted","Data":"423664f6d0c067e86ff3a88d43496ab7e7e585dc2afac593cc472526292328fb"} Oct 30 18:45:36.201623 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:36.201571 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj" podStartSLOduration=2.201550086 podStartE2EDuration="2.201550086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:36.20010651 +0000 UTC m=+696.227211468" watchObservedRunningTime="2026-10-30 18:45:36.201550086 +0000 UTC m=+696.228655046" Oct 30 18:45:37.091377 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:37.091328 4531 generic.go:356] "Generic (PLEG): container finished" podID="dacd0248-6d7e-4835-99dd-7aaa5ce74477" containerID="adcc63ff33ca6bcc86bc0c949842e0a95c02ed885bb4de346bd92b7da525a368" exitCode=0 Oct 30 18:45:37.091865 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:37.091409 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj" event={"ID":"dacd0248-6d7e-4835-99dd-7aaa5ce74477","Type":"ContainerDied","Data":"adcc63ff33ca6bcc86bc0c949842e0a95c02ed885bb4de346bd92b7da525a368"} Oct 30 18:45:38.255692 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:38.255656 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj" Oct 30 18:45:38.346915 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:38.346816 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dacd0248-6d7e-4835-99dd-7aaa5ce74477-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dacd0248-6d7e-4835-99dd-7aaa5ce74477-maas-api-xks-ca\") pod \"dacd0248-6d7e-4835-99dd-7aaa5ce74477\" (UID: \"dacd0248-6d7e-4835-99dd-7aaa5ce74477\") " Oct 30 18:45:38.346915 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:38.346863 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dacd0248-6d7e-4835-99dd-7aaa5ce74477-kube-api-access-mmbgh\" (UniqueName: \"kubernetes.io/projected/dacd0248-6d7e-4835-99dd-7aaa5ce74477-kube-api-access-mmbgh\") pod \"dacd0248-6d7e-4835-99dd-7aaa5ce74477\" (UID: \"dacd0248-6d7e-4835-99dd-7aaa5ce74477\") " Oct 30 18:45:38.346915 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:38.346913 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dacd0248-6d7e-4835-99dd-7aaa5ce74477-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dacd0248-6d7e-4835-99dd-7aaa5ce74477-maas-api-openshift-ca\") pod \"dacd0248-6d7e-4835-99dd-7aaa5ce74477\" (UID: \"dacd0248-6d7e-4835-99dd-7aaa5ce74477\") " Oct 30 18:45:38.347366 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:38.347340 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dacd0248-6d7e-4835-99dd-7aaa5ce74477-maas-api-openshift-ca" pod "dacd0248-6d7e-4835-99dd-7aaa5ce74477" (UID: "dacd0248-6d7e-4835-99dd-7aaa5ce74477"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:38.349303 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:38.349273 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dacd0248-6d7e-4835-99dd-7aaa5ce74477-maas-api-xks-ca" pod "dacd0248-6d7e-4835-99dd-7aaa5ce74477" (UID: "dacd0248-6d7e-4835-99dd-7aaa5ce74477"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:45:38.349440 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:38.349400 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dacd0248-6d7e-4835-99dd-7aaa5ce74477-kube-api-access-mmbgh" pod "dacd0248-6d7e-4835-99dd-7aaa5ce74477" (UID: "dacd0248-6d7e-4835-99dd-7aaa5ce74477"). InnerVolumeSpecName "kube-api-access-mmbgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:38.447891 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:38.447850 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dacd0248-6d7e-4835-99dd-7aaa5ce74477-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:45:38.447891 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:38.447884 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmbgh\" (UniqueName: \"kubernetes.io/projected/dacd0248-6d7e-4835-99dd-7aaa5ce74477-kube-api-access-mmbgh\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:45:38.447891 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:38.447898 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dacd0248-6d7e-4835-99dd-7aaa5ce74477-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:45:39.102691 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:39.102659 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj" Oct 30 18:45:39.102691 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:39.102677 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj" event={"ID":"dacd0248-6d7e-4835-99dd-7aaa5ce74477","Type":"ContainerDied","Data":"423664f6d0c067e86ff3a88d43496ab7e7e585dc2afac593cc472526292328fb"} Oct 30 18:45:39.102876 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:39.102724 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="423664f6d0c067e86ff3a88d43496ab7e7e585dc2afac593cc472526292328fb" Oct 30 18:45:39.855168 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:39.855114 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc"] Oct 30 18:45:39.855576 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:39.855513 4531 cpu_manager.go:414] "containerMap: removing container" podUID="dacd0248-6d7e-4835-99dd-7aaa5ce74477" containerName="revoke-keys" Oct 30 18:45:39.855576 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:39.855525 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dacd0248-6d7e-4835-99dd-7aaa5ce74477" containerName="revoke-keys" Oct 30 18:45:39.855647 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:39.855583 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="dacd0248-6d7e-4835-99dd-7aaa5ce74477" containerName="revoke-keys" Oct 30 18:45:39.858800 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:39.858777 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc" Oct 30 18:45:39.911664 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:39.911627 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:45:39.960896 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:39.960851 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70529d01-abcd-4a0b-804e-a8f0ab0cee1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc\" (UID: \"70529d01-abcd-4a0b-804e-a8f0ab0cee1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc" Oct 30 18:45:39.961104 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:39.960923 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70529d01-abcd-4a0b-804e-a8f0ab0cee1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc\" (UID: \"70529d01-abcd-4a0b-804e-a8f0ab0cee1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc" Oct 30 18:45:39.961104 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:39.960933 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:45:39.961104 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:39.960982 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:45:39.961104 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:39.961015 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:45:39.961104 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:39.961072 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfzzs\" (UniqueName: \"kubernetes.io/projected/70529d01-abcd-4a0b-804e-a8f0ab0cee1f-kube-api-access-lfzzs\") pod \"maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc\" (UID: \"70529d01-abcd-4a0b-804e-a8f0ab0cee1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc" Oct 30 18:45:39.962677 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:39.962657 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc"] Oct 30 18:45:40.062115 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:40.062073 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70529d01-abcd-4a0b-804e-a8f0ab0cee1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc\" (UID: \"70529d01-abcd-4a0b-804e-a8f0ab0cee1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc" Oct 30 18:45:40.062115 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:40.062150 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfzzs\" (UniqueName: \"kubernetes.io/projected/70529d01-abcd-4a0b-804e-a8f0ab0cee1f-kube-api-access-lfzzs\") pod \"maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc\" (UID: \"70529d01-abcd-4a0b-804e-a8f0ab0cee1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc" Oct 30 18:45:40.062368 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:40.062203 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70529d01-abcd-4a0b-804e-a8f0ab0cee1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc\" (UID: \"70529d01-abcd-4a0b-804e-a8f0ab0cee1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc" Oct 30 18:45:40.062855 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:40.062832 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70529d01-abcd-4a0b-804e-a8f0ab0cee1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc\" (UID: \"70529d01-abcd-4a0b-804e-a8f0ab0cee1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc" Oct 30 18:45:40.064767 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:40.064737 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70529d01-abcd-4a0b-804e-a8f0ab0cee1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc\" (UID: \"70529d01-abcd-4a0b-804e-a8f0ab0cee1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc" Oct 30 18:45:40.124333 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:40.124259 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfzzs\" (UniqueName: \"kubernetes.io/projected/70529d01-abcd-4a0b-804e-a8f0ab0cee1f-kube-api-access-lfzzs\") pod \"maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc\" (UID: \"70529d01-abcd-4a0b-804e-a8f0ab0cee1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc" Oct 30 18:45:40.174886 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:40.174857 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc" Oct 30 18:45:40.555847 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:40.555804 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc"] Oct 30 18:45:41.112404 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:41.112303 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc" event={"ID":"70529d01-abcd-4a0b-804e-a8f0ab0cee1f","Type":"ContainerStarted","Data":"cd71bea467af19c3f2cfc2f7ce3b84dd210e7815c611d93f92ed98a49541c45c"} Oct 30 18:45:41.112404 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:41.112363 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc" event={"ID":"70529d01-abcd-4a0b-804e-a8f0ab0cee1f","Type":"ContainerStarted","Data":"b23aaa56d09ca4c9b4af4def8ca967bcb2cc0fd12e074eaeb055d655107abf41"} Oct 30 18:45:41.228958 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:41.228902 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc" podStartSLOduration=2.228886191 podStartE2EDuration="2.228886191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:41.228118711 +0000 UTC m=+701.255223669" watchObservedRunningTime="2026-10-30 18:45:41.228886191 +0000 UTC m=+701.255991149" Oct 30 18:45:42.119182 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:42.119112 4531 generic.go:356] "Generic (PLEG): container finished" podID="70529d01-abcd-4a0b-804e-a8f0ab0cee1f" containerID="cd71bea467af19c3f2cfc2f7ce3b84dd210e7815c611d93f92ed98a49541c45c" exitCode=0 Oct 30 18:45:42.119182 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:42.119164 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc" event={"ID":"70529d01-abcd-4a0b-804e-a8f0ab0cee1f","Type":"ContainerDied","Data":"cd71bea467af19c3f2cfc2f7ce3b84dd210e7815c611d93f92ed98a49541c45c"} Oct 30 18:45:43.268747 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:43.268716 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc" Oct 30 18:45:43.392808 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:43.392715 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70529d01-abcd-4a0b-804e-a8f0ab0cee1f-kube-api-access-lfzzs\" (UniqueName: \"kubernetes.io/projected/70529d01-abcd-4a0b-804e-a8f0ab0cee1f-kube-api-access-lfzzs\") pod \"70529d01-abcd-4a0b-804e-a8f0ab0cee1f\" (UID: \"70529d01-abcd-4a0b-804e-a8f0ab0cee1f\") " Oct 30 18:45:43.392988 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:43.392810 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70529d01-abcd-4a0b-804e-a8f0ab0cee1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70529d01-abcd-4a0b-804e-a8f0ab0cee1f-maas-api-xks-ca\") pod \"70529d01-abcd-4a0b-804e-a8f0ab0cee1f\" (UID: \"70529d01-abcd-4a0b-804e-a8f0ab0cee1f\") " Oct 30 18:45:43.392988 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:43.392857 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70529d01-abcd-4a0b-804e-a8f0ab0cee1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70529d01-abcd-4a0b-804e-a8f0ab0cee1f-maas-api-openshift-ca\") pod \"70529d01-abcd-4a0b-804e-a8f0ab0cee1f\" (UID: \"70529d01-abcd-4a0b-804e-a8f0ab0cee1f\") " Oct 30 18:45:43.393287 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:43.393260 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70529d01-abcd-4a0b-804e-a8f0ab0cee1f-maas-api-openshift-ca" pod "70529d01-abcd-4a0b-804e-a8f0ab0cee1f" (UID: "70529d01-abcd-4a0b-804e-a8f0ab0cee1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:43.395434 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:43.395401 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70529d01-abcd-4a0b-804e-a8f0ab0cee1f-kube-api-access-lfzzs" pod "70529d01-abcd-4a0b-804e-a8f0ab0cee1f" (UID: "70529d01-abcd-4a0b-804e-a8f0ab0cee1f"). InnerVolumeSpecName "kube-api-access-lfzzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:43.395434 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:43.395416 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70529d01-abcd-4a0b-804e-a8f0ab0cee1f-maas-api-xks-ca" pod "70529d01-abcd-4a0b-804e-a8f0ab0cee1f" (UID: "70529d01-abcd-4a0b-804e-a8f0ab0cee1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:45:43.494480 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:43.494431 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfzzs\" (UniqueName: \"kubernetes.io/projected/70529d01-abcd-4a0b-804e-a8f0ab0cee1f-kube-api-access-lfzzs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:45:43.494480 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:43.494469 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70529d01-abcd-4a0b-804e-a8f0ab0cee1f-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:45:43.494480 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:43.494480 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70529d01-abcd-4a0b-804e-a8f0ab0cee1f-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:45:44.129858 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:44.129825 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc" event={"ID":"70529d01-abcd-4a0b-804e-a8f0ab0cee1f","Type":"ContainerDied","Data":"b23aaa56d09ca4c9b4af4def8ca967bcb2cc0fd12e074eaeb055d655107abf41"} Oct 30 18:45:44.129858 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:44.129856 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc" Oct 30 18:45:44.130078 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:44.129874 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b23aaa56d09ca4c9b4af4def8ca967bcb2cc0fd12e074eaeb055d655107abf41" Oct 30 18:45:47.398691 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:47.398647 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a49490-v2htg"] Oct 30 18:45:47.448883 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:47.448846 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a49490-v2htg"] Oct 30 18:45:48.097420 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:48.097381 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f4a27a8-bfc4-440a-996f-e0bce34bdc82" path="/var/lib/kubelet/pods/2f4a27a8-bfc4-440a-996f-e0bce34bdc82/volumes" Oct 30 18:45:49.019559 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:49.019515 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-620f2843-zf7h7"] Oct 30 18:45:49.019959 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:49.019892 4531 cpu_manager.go:414] "containerMap: removing container" podUID="70529d01-abcd-4a0b-804e-a8f0ab0cee1f" containerName="revoke-keys" Oct 30 18:45:49.019959 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:49.019904 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70529d01-abcd-4a0b-804e-a8f0ab0cee1f" containerName="revoke-keys" Oct 30 18:45:49.020050 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:49.019964 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="70529d01-abcd-4a0b-804e-a8f0ab0cee1f" containerName="revoke-keys" Oct 30 18:45:49.023194 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:49.023169 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-620f2843-zf7h7" Oct 30 18:45:49.076062 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:49.076025 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:45:49.094450 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:49.094413 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:45:49.094450 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:49.094416 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:45:49.125434 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:49.125397 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:45:49.126960 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:49.126934 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-620f2843-zf7h7"] Oct 30 18:45:49.149233 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:49.149182 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffeb54d6-e67f-4e63-8167-6bfed3822609-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-620f2843-zf7h7\" (UID: \"ffeb54d6-e67f-4e63-8167-6bfed3822609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-620f2843-zf7h7" Oct 30 18:45:49.149427 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:49.149314 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ccf9\" (UniqueName: \"kubernetes.io/projected/ffeb54d6-e67f-4e63-8167-6bfed3822609-kube-api-access-6ccf9\") pod \"maas-api-revoke-keys-e2e-derive-620f2843-zf7h7\" (UID: \"ffeb54d6-e67f-4e63-8167-6bfed3822609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-620f2843-zf7h7" Oct 30 18:45:49.149427 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:49.149376 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffeb54d6-e67f-4e63-8167-6bfed3822609-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-620f2843-zf7h7\" (UID: \"ffeb54d6-e67f-4e63-8167-6bfed3822609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-620f2843-zf7h7" Oct 30 18:45:49.250262 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:49.250123 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ccf9\" (UniqueName: \"kubernetes.io/projected/ffeb54d6-e67f-4e63-8167-6bfed3822609-kube-api-access-6ccf9\") pod \"maas-api-revoke-keys-e2e-derive-620f2843-zf7h7\" (UID: \"ffeb54d6-e67f-4e63-8167-6bfed3822609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-620f2843-zf7h7" Oct 30 18:45:49.250262 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:49.250257 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffeb54d6-e67f-4e63-8167-6bfed3822609-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-620f2843-zf7h7\" (UID: \"ffeb54d6-e67f-4e63-8167-6bfed3822609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-620f2843-zf7h7" Oct 30 18:45:49.250521 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:49.250334 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffeb54d6-e67f-4e63-8167-6bfed3822609-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-620f2843-zf7h7\" (UID: \"ffeb54d6-e67f-4e63-8167-6bfed3822609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-620f2843-zf7h7" Oct 30 18:45:49.251078 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:49.251050 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffeb54d6-e67f-4e63-8167-6bfed3822609-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-620f2843-zf7h7\" (UID: \"ffeb54d6-e67f-4e63-8167-6bfed3822609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-620f2843-zf7h7" Oct 30 18:45:49.253093 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:49.253064 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffeb54d6-e67f-4e63-8167-6bfed3822609-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-620f2843-zf7h7\" (UID: \"ffeb54d6-e67f-4e63-8167-6bfed3822609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-620f2843-zf7h7" Oct 30 18:45:49.311537 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:49.311448 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ccf9\" (UniqueName: \"kubernetes.io/projected/ffeb54d6-e67f-4e63-8167-6bfed3822609-kube-api-access-6ccf9\") pod \"maas-api-revoke-keys-e2e-derive-620f2843-zf7h7\" (UID: \"ffeb54d6-e67f-4e63-8167-6bfed3822609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-620f2843-zf7h7" Oct 30 18:45:49.337856 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:49.337814 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-620f2843-zf7h7" Oct 30 18:45:49.713175 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:49.713142 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-620f2843-zf7h7"] Oct 30 18:45:49.715596 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:45:49.715561 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffeb54d6_e67f_4e63_8167_6bfed3822609.slice/crio-cb3490d92fb1cb2596ea1a7583a72fa4b56183ee17b3386a15fdc5f83911ee05 WatchSource:0}: Error finding container cb3490d92fb1cb2596ea1a7583a72fa4b56183ee17b3386a15fdc5f83911ee05: Status 404 returned error can't find the container with id cb3490d92fb1cb2596ea1a7583a72fa4b56183ee17b3386a15fdc5f83911ee05 Oct 30 18:45:50.160040 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:50.160000 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-620f2843-zf7h7" event={"ID":"ffeb54d6-e67f-4e63-8167-6bfed3822609","Type":"ContainerStarted","Data":"fd101a80fc008ee6357cc696e58c83dfe27e3bb4609fa134095e46e9104658b7"} Oct 30 18:45:50.160531 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:50.160064 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-620f2843-zf7h7" event={"ID":"ffeb54d6-e67f-4e63-8167-6bfed3822609","Type":"ContainerStarted","Data":"cb3490d92fb1cb2596ea1a7583a72fa4b56183ee17b3386a15fdc5f83911ee05"} Oct 30 18:45:50.275080 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:50.274958 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-620f2843-zf7h7" podStartSLOduration=2.274941406 podStartE2EDuration="2.274941406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:50.274272485 +0000 UTC m=+710.301377445" watchObservedRunningTime="2026-10-30 18:45:50.274941406 +0000 UTC m=+710.302046365" Oct 30 18:45:51.165699 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:51.165661 4531 generic.go:356] "Generic (PLEG): container finished" podID="ffeb54d6-e67f-4e63-8167-6bfed3822609" containerID="fd101a80fc008ee6357cc696e58c83dfe27e3bb4609fa134095e46e9104658b7" exitCode=0 Oct 30 18:45:51.166153 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:51.165745 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-620f2843-zf7h7" event={"ID":"ffeb54d6-e67f-4e63-8167-6bfed3822609","Type":"ContainerDied","Data":"fd101a80fc008ee6357cc696e58c83dfe27e3bb4609fa134095e46e9104658b7"} Oct 30 18:45:52.317636 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:52.317607 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-620f2843-zf7h7" Oct 30 18:45:52.378949 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:52.378906 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffeb54d6-e67f-4e63-8167-6bfed3822609-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffeb54d6-e67f-4e63-8167-6bfed3822609-maas-api-xks-ca\") pod \"ffeb54d6-e67f-4e63-8167-6bfed3822609\" (UID: \"ffeb54d6-e67f-4e63-8167-6bfed3822609\") " Oct 30 18:45:52.379166 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:52.378968 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffeb54d6-e67f-4e63-8167-6bfed3822609-kube-api-access-6ccf9\" (UniqueName: \"kubernetes.io/projected/ffeb54d6-e67f-4e63-8167-6bfed3822609-kube-api-access-6ccf9\") pod \"ffeb54d6-e67f-4e63-8167-6bfed3822609\" (UID: \"ffeb54d6-e67f-4e63-8167-6bfed3822609\") " Oct 30 18:45:52.379166 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:52.379021 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffeb54d6-e67f-4e63-8167-6bfed3822609-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffeb54d6-e67f-4e63-8167-6bfed3822609-maas-api-openshift-ca\") pod \"ffeb54d6-e67f-4e63-8167-6bfed3822609\" (UID: \"ffeb54d6-e67f-4e63-8167-6bfed3822609\") " Oct 30 18:45:52.379413 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:52.379388 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffeb54d6-e67f-4e63-8167-6bfed3822609-maas-api-openshift-ca" pod "ffeb54d6-e67f-4e63-8167-6bfed3822609" (UID: "ffeb54d6-e67f-4e63-8167-6bfed3822609"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:52.387422 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:52.381477 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffeb54d6-e67f-4e63-8167-6bfed3822609-maas-api-xks-ca" pod "ffeb54d6-e67f-4e63-8167-6bfed3822609" (UID: "ffeb54d6-e67f-4e63-8167-6bfed3822609"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:45:52.387644 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:52.387591 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffeb54d6-e67f-4e63-8167-6bfed3822609-kube-api-access-6ccf9" pod "ffeb54d6-e67f-4e63-8167-6bfed3822609" (UID: "ffeb54d6-e67f-4e63-8167-6bfed3822609"). InnerVolumeSpecName "kube-api-access-6ccf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:52.480551 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:52.480447 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffeb54d6-e67f-4e63-8167-6bfed3822609-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:45:52.480551 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:52.480489 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffeb54d6-e67f-4e63-8167-6bfed3822609-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:45:52.480551 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:52.480499 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ccf9\" (UniqueName: \"kubernetes.io/projected/ffeb54d6-e67f-4e63-8167-6bfed3822609-kube-api-access-6ccf9\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:45:53.177619 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:53.177571 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-620f2843-zf7h7" event={"ID":"ffeb54d6-e67f-4e63-8167-6bfed3822609","Type":"ContainerDied","Data":"cb3490d92fb1cb2596ea1a7583a72fa4b56183ee17b3386a15fdc5f83911ee05"} Oct 30 18:45:53.177619 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:53.177598 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-620f2843-zf7h7" Oct 30 18:45:53.177861 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:53.177631 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb3490d92fb1cb2596ea1a7583a72fa4b56183ee17b3386a15fdc5f83911ee05" Oct 30 18:45:56.400840 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:45:56.400798 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:45:57.246905 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:57.246866 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj"] Oct 30 18:45:57.297055 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:57.297011 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-59144a1b-mvtkj"] Oct 30 18:45:58.098162 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:45:58.098109 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dacd0248-6d7e-4835-99dd-7aaa5ce74477" path="/var/lib/kubelet/pods/dacd0248-6d7e-4835-99dd-7aaa5ce74477/volumes" Oct 30 18:46:00.670243 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:00.670198 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp"] Oct 30 18:46:05.670633 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:05.670550 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp" podUID="0a907a3e-3c6d-4a08-9a91-4d9f6dc12891" containerName="payload-processing" containerID="cri-o://9a31de07d5f3838de324454e7f119f5ddcb055b261fe61140dc457373713fc40" gracePeriod=25 Oct 30 18:46:05.929964 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:05.929885 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp" Oct 30 18:46:06.006840 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:06.006803 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a907a3e-3c6d-4a08-9a91-4d9f6dc12891-kube-api-access-rsz7b\" (UniqueName: \"kubernetes.io/projected/0a907a3e-3c6d-4a08-9a91-4d9f6dc12891-kube-api-access-rsz7b\") pod \"0a907a3e-3c6d-4a08-9a91-4d9f6dc12891\" (UID: \"0a907a3e-3c6d-4a08-9a91-4d9f6dc12891\") " Oct 30 18:46:06.007021 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:06.006885 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a907a3e-3c6d-4a08-9a91-4d9f6dc12891-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a907a3e-3c6d-4a08-9a91-4d9f6dc12891-plugins-config-volume\") pod \"0a907a3e-3c6d-4a08-9a91-4d9f6dc12891\" (UID: \"0a907a3e-3c6d-4a08-9a91-4d9f6dc12891\") " Oct 30 18:46:06.007348 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:06.007318 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a907a3e-3c6d-4a08-9a91-4d9f6dc12891-plugins-config-volume" pod "0a907a3e-3c6d-4a08-9a91-4d9f6dc12891" (UID: "0a907a3e-3c6d-4a08-9a91-4d9f6dc12891"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:06.009373 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:06.009321 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a907a3e-3c6d-4a08-9a91-4d9f6dc12891-kube-api-access-rsz7b" pod "0a907a3e-3c6d-4a08-9a91-4d9f6dc12891" (UID: "0a907a3e-3c6d-4a08-9a91-4d9f6dc12891"). InnerVolumeSpecName "kube-api-access-rsz7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:06.107596 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:06.107562 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsz7b\" (UniqueName: \"kubernetes.io/projected/0a907a3e-3c6d-4a08-9a91-4d9f6dc12891-kube-api-access-rsz7b\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:46:06.107596 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:06.107593 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a907a3e-3c6d-4a08-9a91-4d9f6dc12891-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:46:06.240835 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:06.240745 4531 generic.go:356] "Generic (PLEG): container finished" podID="0a907a3e-3c6d-4a08-9a91-4d9f6dc12891" containerID="9a31de07d5f3838de324454e7f119f5ddcb055b261fe61140dc457373713fc40" exitCode=0 Oct 30 18:46:06.240835 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:06.240791 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp" Oct 30 18:46:06.241028 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:06.240830 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp" event={"ID":"0a907a3e-3c6d-4a08-9a91-4d9f6dc12891","Type":"ContainerDied","Data":"9a31de07d5f3838de324454e7f119f5ddcb055b261fe61140dc457373713fc40"} Oct 30 18:46:06.241028 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:06.240886 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp" event={"ID":"0a907a3e-3c6d-4a08-9a91-4d9f6dc12891","Type":"ContainerDied","Data":"82da28344299b3a6e50a5dad5d5cbd1ae373d07ed93ff81c4f2941e086cba1a9"} Oct 30 18:46:06.241028 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:06.240908 4531 scope.go:122] "RemoveContainer" containerID="9a31de07d5f3838de324454e7f119f5ddcb055b261fe61140dc457373713fc40" Oct 30 18:46:06.250368 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:06.250329 4531 scope.go:122] "RemoveContainer" containerID="9a31de07d5f3838de324454e7f119f5ddcb055b261fe61140dc457373713fc40" Oct 30 18:46:06.250671 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:46:06.250651 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a31de07d5f3838de324454e7f119f5ddcb055b261fe61140dc457373713fc40\": container with ID starting with 9a31de07d5f3838de324454e7f119f5ddcb055b261fe61140dc457373713fc40 not found: ID does not exist" containerID="9a31de07d5f3838de324454e7f119f5ddcb055b261fe61140dc457373713fc40" Oct 30 18:46:06.250744 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:06.250682 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a31de07d5f3838de324454e7f119f5ddcb055b261fe61140dc457373713fc40"} err="failed to get container status \"9a31de07d5f3838de324454e7f119f5ddcb055b261fe61140dc457373713fc40\": rpc error: code = NotFound desc = could not find container \"9a31de07d5f3838de324454e7f119f5ddcb055b261fe61140dc457373713fc40\": container with ID starting with 9a31de07d5f3838de324454e7f119f5ddcb055b261fe61140dc457373713fc40 not found: ID does not exist" Oct 30 18:46:06.403776 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:06.403734 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp"] Oct 30 18:46:06.408474 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:06.408430 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-620f2843-7bbcd6c7d6-d9ddp"] Oct 30 18:46:07.985855 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:07.985810 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc"] Oct 30 18:46:08.035330 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:08.035289 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-529e04e2-cf8gc"] Oct 30 18:46:08.097695 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:08.097663 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a907a3e-3c6d-4a08-9a91-4d9f6dc12891" path="/var/lib/kubelet/pods/0a907a3e-3c6d-4a08-9a91-4d9f6dc12891/volumes" Oct 30 18:46:08.098062 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:08.098048 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70529d01-abcd-4a0b-804e-a8f0ab0cee1f" path="/var/lib/kubelet/pods/70529d01-abcd-4a0b-804e-a8f0ab0cee1f/volumes" Oct 30 18:46:10.104085 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:10.104052 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v"] Oct 30 18:46:10.104504 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:10.104396 4531 cpu_manager.go:414] "containerMap: removing container" podUID="0a907a3e-3c6d-4a08-9a91-4d9f6dc12891" containerName="payload-processing" Oct 30 18:46:10.104504 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:10.104408 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a907a3e-3c6d-4a08-9a91-4d9f6dc12891" containerName="payload-processing" Oct 30 18:46:10.104504 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:10.104429 4531 cpu_manager.go:414] "containerMap: removing container" podUID="ffeb54d6-e67f-4e63-8167-6bfed3822609" containerName="revoke-keys" Oct 30 18:46:10.104504 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:10.104434 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffeb54d6-e67f-4e63-8167-6bfed3822609" containerName="revoke-keys" Oct 30 18:46:10.104681 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:10.104511 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffeb54d6-e67f-4e63-8167-6bfed3822609" containerName="revoke-keys" Oct 30 18:46:10.104681 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:10.104523 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a907a3e-3c6d-4a08-9a91-4d9f6dc12891" containerName="payload-processing" Oct 30 18:46:10.108991 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:10.108962 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v" Oct 30 18:46:10.155327 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:10.155292 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v"] Oct 30 18:46:10.203291 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:10.203261 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-4a7a52\"" Oct 30 18:46:10.203472 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:10.203311 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-4a7a52-dockercfg-zbh2k\"" Oct 30 18:46:10.245299 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:10.245258 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx7wv\" (UniqueName: \"kubernetes.io/projected/06335665-17e4-4b6a-b7a6-30ef62625cfe-kube-api-access-dx7wv\") pod \"payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v\" (UID: \"06335665-17e4-4b6a-b7a6-30ef62625cfe\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v" Oct 30 18:46:10.245481 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:10.245304 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06335665-17e4-4b6a-b7a6-30ef62625cfe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v\" (UID: \"06335665-17e4-4b6a-b7a6-30ef62625cfe\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v" Oct 30 18:46:10.345750 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:10.345716 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06335665-17e4-4b6a-b7a6-30ef62625cfe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v\" (UID: \"06335665-17e4-4b6a-b7a6-30ef62625cfe\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v" Oct 30 18:46:10.345960 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:10.345841 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx7wv\" (UniqueName: \"kubernetes.io/projected/06335665-17e4-4b6a-b7a6-30ef62625cfe-kube-api-access-dx7wv\") pod \"payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v\" (UID: \"06335665-17e4-4b6a-b7a6-30ef62625cfe\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v" Oct 30 18:46:10.346591 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:10.346565 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06335665-17e4-4b6a-b7a6-30ef62625cfe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v\" (UID: \"06335665-17e4-4b6a-b7a6-30ef62625cfe\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v" Oct 30 18:46:10.407174 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:10.407047 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx7wv\" (UniqueName: \"kubernetes.io/projected/06335665-17e4-4b6a-b7a6-30ef62625cfe-kube-api-access-dx7wv\") pod \"payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v\" (UID: \"06335665-17e4-4b6a-b7a6-30ef62625cfe\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v" Oct 30 18:46:10.421753 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:10.421713 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v" Oct 30 18:46:10.796988 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:10.796952 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v"] Oct 30 18:46:10.800885 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:46:10.800844 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06335665_17e4_4b6a_b7a6_30ef62625cfe.slice/crio-e45aed8b09b8887278f151d3c4addfeafbee87affb77b64d56721ca0dddd69ff WatchSource:0}: Error finding container e45aed8b09b8887278f151d3c4addfeafbee87affb77b64d56721ca0dddd69ff: Status 404 returned error can't find the container with id e45aed8b09b8887278f151d3c4addfeafbee87affb77b64d56721ca0dddd69ff Oct 30 18:46:11.266409 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:11.266368 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v" event={"ID":"06335665-17e4-4b6a-b7a6-30ef62625cfe","Type":"ContainerStarted","Data":"4163e63fc110f331c0d6d095431820d6ae72f53e474a82e0df07ce10498e43a5"} Oct 30 18:46:11.266881 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:11.266426 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v" event={"ID":"06335665-17e4-4b6a-b7a6-30ef62625cfe","Type":"ContainerStarted","Data":"e45aed8b09b8887278f151d3c4addfeafbee87affb77b64d56721ca0dddd69ff"} Oct 30 18:46:11.266881 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:11.266533 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v" Oct 30 18:46:11.383851 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:11.383788 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v" podStartSLOduration=1.383771867 podStartE2EDuration="1.383771867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:11.382295907 +0000 UTC m=+731.409400871" watchObservedRunningTime="2026-10-30 18:46:11.383771867 +0000 UTC m=+731.410876825" Oct 30 18:46:22.274102 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:22.274067 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v" Oct 30 18:46:23.257199 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:23.257156 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-620f2843-zf7h7"] Oct 30 18:46:23.259751 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:23.259715 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-620f2843-zf7h7"] Oct 30 18:46:24.097612 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:24.097570 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffeb54d6-e67f-4e63-8167-6bfed3822609" path="/var/lib/kubelet/pods/ffeb54d6-e67f-4e63-8167-6bfed3822609/volumes" Oct 30 18:46:26.399093 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:46:26.399052 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:46:44.016654 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.016560 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4"] Oct 30 18:46:44.020328 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.020304 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 18:46:44.051837 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.051802 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-model-cache\") pod \"simulator-w3-3b6013-kserve-74476cc78c-kvzz4\" (UID: \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\") " pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 18:46:44.052114 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.052093 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-tmp-dir\") pod \"simulator-w3-3b6013-kserve-74476cc78c-kvzz4\" (UID: \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\") " pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 18:46:44.052362 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.052339 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4r5d\" (UniqueName: \"kubernetes.io/projected/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-kube-api-access-l4r5d\") pod \"simulator-w3-3b6013-kserve-74476cc78c-kvzz4\" (UID: \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\") " pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 18:46:44.052512 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.052493 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-tls-certs\") pod \"simulator-w3-3b6013-kserve-74476cc78c-kvzz4\" (UID: \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\") " pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 18:46:44.052717 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.052664 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-home\") pod \"simulator-w3-3b6013-kserve-74476cc78c-kvzz4\" (UID: \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\") " pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 18:46:44.053028 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.053010 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-dshm\") pod \"simulator-w3-3b6013-kserve-74476cc78c-kvzz4\" (UID: \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\") " pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 18:46:44.072593 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.072560 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-3b6013\"/\"kube-root-ca.crt\"" Oct 30 18:46:44.122042 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.122000 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-3b6013\"/\"openshift-service-ca.crt\"" Oct 30 18:46:44.122042 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.122006 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3b6013\"/\"default-dockercfg-dfsp2\"" Oct 30 18:46:44.122281 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.122006 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3b6013\"/\"simulator-w3-3b6013-kserve-self-signed-certs\"" Oct 30 18:46:44.123588 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.123566 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4"] Oct 30 18:46:44.153752 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.153712 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4r5d\" (UniqueName: \"kubernetes.io/projected/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-kube-api-access-l4r5d\") pod \"simulator-w3-3b6013-kserve-74476cc78c-kvzz4\" (UID: \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\") " pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 18:46:44.153957 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.153779 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-tls-certs\") pod \"simulator-w3-3b6013-kserve-74476cc78c-kvzz4\" (UID: \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\") " pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 18:46:44.153957 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.153901 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-home\") pod \"simulator-w3-3b6013-kserve-74476cc78c-kvzz4\" (UID: \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\") " pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 18:46:44.153957 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.153939 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-dshm\") pod \"simulator-w3-3b6013-kserve-74476cc78c-kvzz4\" (UID: \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\") " pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 18:46:44.154078 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.153990 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-model-cache\") pod \"simulator-w3-3b6013-kserve-74476cc78c-kvzz4\" (UID: \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\") " pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 18:46:44.154078 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.154015 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-tmp-dir\") pod \"simulator-w3-3b6013-kserve-74476cc78c-kvzz4\" (UID: \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\") " pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 18:46:44.154498 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.154468 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-home\") pod \"simulator-w3-3b6013-kserve-74476cc78c-kvzz4\" (UID: \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\") " pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 18:46:44.154767 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.154511 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-model-cache\") pod \"simulator-w3-3b6013-kserve-74476cc78c-kvzz4\" (UID: \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\") " pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 18:46:44.154767 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.154589 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-tmp-dir\") pod \"simulator-w3-3b6013-kserve-74476cc78c-kvzz4\" (UID: \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\") " pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 18:46:44.156458 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.156419 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-dshm\") pod \"simulator-w3-3b6013-kserve-74476cc78c-kvzz4\" (UID: \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\") " pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 18:46:44.156837 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.156817 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-tls-certs\") pod \"simulator-w3-3b6013-kserve-74476cc78c-kvzz4\" (UID: \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\") " pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 18:46:44.214959 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.214926 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4r5d\" (UniqueName: \"kubernetes.io/projected/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-kube-api-access-l4r5d\") pod \"simulator-w3-3b6013-kserve-74476cc78c-kvzz4\" (UID: \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\") " pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 18:46:44.354581 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.354490 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 18:46:44.732585 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:44.732383 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4"] Oct 30 18:46:44.735431 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:46:44.735405 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b32fdf8_e824_4cc6_b25b_ef5f1ed2d630.slice/crio-3d26c25d0015a77c69d07f92e2e855ab481a3aa6659f85ed9d867c2748022bff WatchSource:0}: Error finding container 3d26c25d0015a77c69d07f92e2e855ab481a3aa6659f85ed9d867c2748022bff: Status 404 returned error can't find the container with id 3d26c25d0015a77c69d07f92e2e855ab481a3aa6659f85ed9d867c2748022bff Oct 30 18:46:45.447352 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:45.447312 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" event={"ID":"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630","Type":"ContainerStarted","Data":"3d26c25d0015a77c69d07f92e2e855ab481a3aa6659f85ed9d867c2748022bff"} Oct 30 18:46:47.459838 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:47.459797 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" event={"ID":"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630","Type":"ContainerStarted","Data":"ade2531a8928518ceaf2b4926c679e0606e48fa045224293bbe51f10610c682c"} Oct 30 18:46:47.577152 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:47.577081 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" podStartSLOduration=2.355771544 podStartE2EDuration="4.577066992s" totalImagesPullingTime="2.221295448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:47.576551386 +0000 UTC m=+767.603656346" watchObservedRunningTime="2026-10-30 18:46:47.577066992 +0000 UTC m=+767.604171951" Oct 30 18:46:48.355279 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:48.355232 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 18:46:54.355718 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:54.355622 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 18:46:54.368449 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:54.368419 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 18:46:54.504787 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:46:54.504754 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 18:46:56.398867 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:46:56.398834 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:47:06.081073 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.081035 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk"] Oct 30 18:47:06.085794 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.085766 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 18:47:06.138178 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.138141 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2ade09\"/\"kube-root-ca.crt\"" Oct 30 18:47:06.157763 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.157721 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-home\") pod \"simulator-w0-2ade09-kserve-6d4d48958d-n9tqk\" (UID: \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\") " pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 18:47:06.157763 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.157765 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf94x\" (UniqueName: \"kubernetes.io/projected/586f370d-9916-43a9-ac7d-e77cc3fcb92f-kube-api-access-nf94x\") pod \"simulator-w0-2ade09-kserve-6d4d48958d-n9tqk\" (UID: \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\") " pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 18:47:06.157976 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.157789 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-dshm\") pod \"simulator-w0-2ade09-kserve-6d4d48958d-n9tqk\" (UID: \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\") " pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 18:47:06.157976 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.157884 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-tmp-dir\") pod \"simulator-w0-2ade09-kserve-6d4d48958d-n9tqk\" (UID: \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\") " pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 18:47:06.158046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.158016 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-model-cache\") pod \"simulator-w0-2ade09-kserve-6d4d48958d-n9tqk\" (UID: \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\") " pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 18:47:06.158080 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.158057 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/586f370d-9916-43a9-ac7d-e77cc3fcb92f-tls-certs\") pod \"simulator-w0-2ade09-kserve-6d4d48958d-n9tqk\" (UID: \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\") " pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 18:47:06.187198 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.187157 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2ade09\"/\"openshift-service-ca.crt\"" Oct 30 18:47:06.187365 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.187211 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2ade09\"/\"simulator-w0-2ade09-kserve-self-signed-certs\"" Oct 30 18:47:06.187365 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.187217 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2ade09\"/\"default-dockercfg-mgqgf\"" Oct 30 18:47:06.188983 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.188963 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk"] Oct 30 18:47:06.259047 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.259000 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf94x\" (UniqueName: \"kubernetes.io/projected/586f370d-9916-43a9-ac7d-e77cc3fcb92f-kube-api-access-nf94x\") pod \"simulator-w0-2ade09-kserve-6d4d48958d-n9tqk\" (UID: \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\") " pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 18:47:06.259047 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.259044 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-dshm\") pod \"simulator-w0-2ade09-kserve-6d4d48958d-n9tqk\" (UID: \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\") " pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 18:47:06.259333 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.259074 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-tmp-dir\") pod \"simulator-w0-2ade09-kserve-6d4d48958d-n9tqk\" (UID: \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\") " pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 18:47:06.259333 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.259156 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-model-cache\") pod \"simulator-w0-2ade09-kserve-6d4d48958d-n9tqk\" (UID: \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\") " pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 18:47:06.259333 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.259281 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/586f370d-9916-43a9-ac7d-e77cc3fcb92f-tls-certs\") pod \"simulator-w0-2ade09-kserve-6d4d48958d-n9tqk\" (UID: \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\") " pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 18:47:06.259572 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.259386 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-home\") pod \"simulator-w0-2ade09-kserve-6d4d48958d-n9tqk\" (UID: \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\") " pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 18:47:06.259572 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.259551 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-model-cache\") pod \"simulator-w0-2ade09-kserve-6d4d48958d-n9tqk\" (UID: \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\") " pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 18:47:06.259643 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.259627 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-tmp-dir\") pod \"simulator-w0-2ade09-kserve-6d4d48958d-n9tqk\" (UID: \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\") " pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 18:47:06.259738 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.259719 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-home\") pod \"simulator-w0-2ade09-kserve-6d4d48958d-n9tqk\" (UID: \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\") " pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 18:47:06.261693 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.261664 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-dshm\") pod \"simulator-w0-2ade09-kserve-6d4d48958d-n9tqk\" (UID: \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\") " pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 18:47:06.261994 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.261974 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/586f370d-9916-43a9-ac7d-e77cc3fcb92f-tls-certs\") pod \"simulator-w0-2ade09-kserve-6d4d48958d-n9tqk\" (UID: \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\") " pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 18:47:06.320633 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.320591 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf94x\" (UniqueName: \"kubernetes.io/projected/586f370d-9916-43a9-ac7d-e77cc3fcb92f-kube-api-access-nf94x\") pod \"simulator-w0-2ade09-kserve-6d4d48958d-n9tqk\" (UID: \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\") " pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 18:47:06.397914 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.397813 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 18:47:06.777685 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:06.777645 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk"] Oct 30 18:47:06.780857 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:47:06.780833 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod586f370d_9916_43a9_ac7d_e77cc3fcb92f.slice/crio-7ce9530da13fa1082019675f6066946fd4aeeee91dad6a2bd45b27c0f85ae516 WatchSource:0}: Error finding container 7ce9530da13fa1082019675f6066946fd4aeeee91dad6a2bd45b27c0f85ae516: Status 404 returned error can't find the container with id 7ce9530da13fa1082019675f6066946fd4aeeee91dad6a2bd45b27c0f85ae516 Oct 30 18:47:07.555493 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:07.555449 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" event={"ID":"586f370d-9916-43a9-ac7d-e77cc3fcb92f","Type":"ContainerStarted","Data":"3cbc4c5ffbf8572bf9fdff80aa73ebaca28dc4ea0934d2aebe78331cf02556dc"} Oct 30 18:47:07.555493 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:07.555496 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" event={"ID":"586f370d-9916-43a9-ac7d-e77cc3fcb92f","Type":"ContainerStarted","Data":"7ce9530da13fa1082019675f6066946fd4aeeee91dad6a2bd45b27c0f85ae516"} Oct 30 18:47:07.676932 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:07.676864 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" podStartSLOduration=1.676843598 podStartE2EDuration="1.676843598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:07.675768354 +0000 UTC m=+787.702873314" watchObservedRunningTime="2026-10-30 18:47:07.676843598 +0000 UTC m=+787.703948558" Oct 30 18:47:08.398069 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:08.398029 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 18:47:13.232548 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.232501 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj"] Oct 30 18:47:13.236093 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.236064 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 18:47:13.332772 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.332730 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-dshm\") pod \"simulator-w2-4a7a52-kserve-575579b57-nzxcj\" (UID: \"678454ff-4761-4bd4-b2a3-2b431bca7934\") " pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 18:47:13.332772 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.332772 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j86jt\" (UniqueName: \"kubernetes.io/projected/678454ff-4761-4bd4-b2a3-2b431bca7934-kube-api-access-j86jt\") pod \"simulator-w2-4a7a52-kserve-575579b57-nzxcj\" (UID: \"678454ff-4761-4bd4-b2a3-2b431bca7934\") " pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 18:47:13.332994 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.332795 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-home\") pod \"simulator-w2-4a7a52-kserve-575579b57-nzxcj\" (UID: \"678454ff-4761-4bd4-b2a3-2b431bca7934\") " pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 18:47:13.332994 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.332867 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-tmp-dir\") pod \"simulator-w2-4a7a52-kserve-575579b57-nzxcj\" (UID: \"678454ff-4761-4bd4-b2a3-2b431bca7934\") " pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 18:47:13.332994 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.332951 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-model-cache\") pod \"simulator-w2-4a7a52-kserve-575579b57-nzxcj\" (UID: \"678454ff-4761-4bd4-b2a3-2b431bca7934\") " pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 18:47:13.332994 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.332973 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/678454ff-4761-4bd4-b2a3-2b431bca7934-tls-certs\") pod \"simulator-w2-4a7a52-kserve-575579b57-nzxcj\" (UID: \"678454ff-4761-4bd4-b2a3-2b431bca7934\") " pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 18:47:13.337504 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.337469 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-4a7a52\"/\"openshift-service-ca.crt\"" Oct 30 18:47:13.337595 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.337469 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-4a7a52\"/\"kube-root-ca.crt\"" Oct 30 18:47:13.337595 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.337537 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-4a7a52\"/\"default-dockercfg-qzpjw\"" Oct 30 18:47:13.337595 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.337534 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-4a7a52\"/\"simulator-w2-4a7a52-kserve-self-signed-certs\"" Oct 30 18:47:13.339344 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.339325 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj"] Oct 30 18:47:13.433790 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.433759 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-tmp-dir\") pod \"simulator-w2-4a7a52-kserve-575579b57-nzxcj\" (UID: \"678454ff-4761-4bd4-b2a3-2b431bca7934\") " pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 18:47:13.433994 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.433820 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-model-cache\") pod \"simulator-w2-4a7a52-kserve-575579b57-nzxcj\" (UID: \"678454ff-4761-4bd4-b2a3-2b431bca7934\") " pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 18:47:13.433994 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.433957 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/678454ff-4761-4bd4-b2a3-2b431bca7934-tls-certs\") pod \"simulator-w2-4a7a52-kserve-575579b57-nzxcj\" (UID: \"678454ff-4761-4bd4-b2a3-2b431bca7934\") " pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 18:47:13.434148 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.434108 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-dshm\") pod \"simulator-w2-4a7a52-kserve-575579b57-nzxcj\" (UID: \"678454ff-4761-4bd4-b2a3-2b431bca7934\") " pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 18:47:13.434203 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.434164 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j86jt\" (UniqueName: \"kubernetes.io/projected/678454ff-4761-4bd4-b2a3-2b431bca7934-kube-api-access-j86jt\") pod \"simulator-w2-4a7a52-kserve-575579b57-nzxcj\" (UID: \"678454ff-4761-4bd4-b2a3-2b431bca7934\") " pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 18:47:13.434256 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.434199 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-home\") pod \"simulator-w2-4a7a52-kserve-575579b57-nzxcj\" (UID: \"678454ff-4761-4bd4-b2a3-2b431bca7934\") " pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 18:47:13.434256 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.434232 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-tmp-dir\") pod \"simulator-w2-4a7a52-kserve-575579b57-nzxcj\" (UID: \"678454ff-4761-4bd4-b2a3-2b431bca7934\") " pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 18:47:13.434332 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.434283 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-model-cache\") pod \"simulator-w2-4a7a52-kserve-575579b57-nzxcj\" (UID: \"678454ff-4761-4bd4-b2a3-2b431bca7934\") " pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 18:47:13.434546 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.434522 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-home\") pod \"simulator-w2-4a7a52-kserve-575579b57-nzxcj\" (UID: \"678454ff-4761-4bd4-b2a3-2b431bca7934\") " pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 18:47:13.436611 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.436582 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-dshm\") pod \"simulator-w2-4a7a52-kserve-575579b57-nzxcj\" (UID: \"678454ff-4761-4bd4-b2a3-2b431bca7934\") " pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 18:47:13.436815 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.436799 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/678454ff-4761-4bd4-b2a3-2b431bca7934-tls-certs\") pod \"simulator-w2-4a7a52-kserve-575579b57-nzxcj\" (UID: \"678454ff-4761-4bd4-b2a3-2b431bca7934\") " pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 18:47:13.497298 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.497267 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j86jt\" (UniqueName: \"kubernetes.io/projected/678454ff-4761-4bd4-b2a3-2b431bca7934-kube-api-access-j86jt\") pod \"simulator-w2-4a7a52-kserve-575579b57-nzxcj\" (UID: \"678454ff-4761-4bd4-b2a3-2b431bca7934\") " pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 18:47:13.549489 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.549440 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 18:47:13.932784 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:13.932760 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj"] Oct 30 18:47:13.936298 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:47:13.936263 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod678454ff_4761_4bd4_b2a3_2b431bca7934.slice/crio-758b41ef548f12d5938c3d35cd43697258b8dcca0015e8a59d4e4e1701a8422e WatchSource:0}: Error finding container 758b41ef548f12d5938c3d35cd43697258b8dcca0015e8a59d4e4e1701a8422e: Status 404 returned error can't find the container with id 758b41ef548f12d5938c3d35cd43697258b8dcca0015e8a59d4e4e1701a8422e Oct 30 18:47:14.589424 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:14.589386 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" event={"ID":"678454ff-4761-4bd4-b2a3-2b431bca7934","Type":"ContainerStarted","Data":"e882a37b69b67c8744ed27e70afaf53f06c0d39d1bc7245cadd52111c312db60"} Oct 30 18:47:14.589424 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:14.589429 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" event={"ID":"678454ff-4761-4bd4-b2a3-2b431bca7934","Type":"ContainerStarted","Data":"758b41ef548f12d5938c3d35cd43697258b8dcca0015e8a59d4e4e1701a8422e"} Oct 30 18:47:14.707253 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:14.707192 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" podStartSLOduration=1.7071734360000002 podStartE2EDuration="1.707173436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:14.705851331 +0000 UTC m=+794.732956291" watchObservedRunningTime="2026-10-30 18:47:14.707173436 +0000 UTC m=+794.734278392" Oct 30 18:47:15.549711 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:15.549661 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 18:47:16.398883 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:16.398833 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 18:47:16.411715 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:16.411682 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 18:47:16.610151 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:16.610107 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 18:47:23.549996 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:23.549947 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 18:47:23.563065 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:23.563031 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 18:47:23.643362 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:23.643327 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 18:47:26.399314 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:47:26.399277 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:47:56.162241 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.162195 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97"] Oct 30 18:47:56.165836 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.165807 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 18:47:56.220388 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.220351 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-fd61e3\"/\"kube-root-ca.crt\"" Oct 30 18:47:56.234728 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.234676 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-dshm\") pod \"simulator-w1-fd61e3-kserve-649c45966f-hrl97\" (UID: \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\") " pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 18:47:56.234728 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.234728 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-tmp-dir\") pod \"simulator-w1-fd61e3-kserve-649c45966f-hrl97\" (UID: \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\") " pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 18:47:56.234944 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.234798 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-model-cache\") pod \"simulator-w1-fd61e3-kserve-649c45966f-hrl97\" (UID: \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\") " pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 18:47:56.234944 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.234823 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-home\") pod \"simulator-w1-fd61e3-kserve-649c45966f-hrl97\" (UID: \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\") " pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 18:47:56.235011 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.234962 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65r8x\" (UniqueName: \"kubernetes.io/projected/1a5f18bf-96ba-4edf-a758-206a1bd926a3-kube-api-access-65r8x\") pod \"simulator-w1-fd61e3-kserve-649c45966f-hrl97\" (UID: \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\") " pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 18:47:56.235011 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.234997 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a5f18bf-96ba-4edf-a758-206a1bd926a3-tls-certs\") pod \"simulator-w1-fd61e3-kserve-649c45966f-hrl97\" (UID: \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\") " pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 18:47:56.269641 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.269583 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-fd61e3\"/\"openshift-service-ca.crt\"" Oct 30 18:47:56.269862 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.269805 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fd61e3\"/\"default-dockercfg-rgk5f\"" Oct 30 18:47:56.269862 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.269840 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fd61e3\"/\"simulator-w1-fd61e3-kserve-self-signed-certs\"" Oct 30 18:47:56.273101 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.273058 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97"] Oct 30 18:47:56.335798 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.335748 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-dshm\") pod \"simulator-w1-fd61e3-kserve-649c45966f-hrl97\" (UID: \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\") " pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 18:47:56.335798 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.335804 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-tmp-dir\") pod \"simulator-w1-fd61e3-kserve-649c45966f-hrl97\" (UID: \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\") " pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 18:47:56.336053 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.335833 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-model-cache\") pod \"simulator-w1-fd61e3-kserve-649c45966f-hrl97\" (UID: \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\") " pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 18:47:56.336053 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.335869 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-home\") pod \"simulator-w1-fd61e3-kserve-649c45966f-hrl97\" (UID: \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\") " pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 18:47:56.336053 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.335952 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65r8x\" (UniqueName: \"kubernetes.io/projected/1a5f18bf-96ba-4edf-a758-206a1bd926a3-kube-api-access-65r8x\") pod \"simulator-w1-fd61e3-kserve-649c45966f-hrl97\" (UID: \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\") " pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 18:47:56.336053 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.335979 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a5f18bf-96ba-4edf-a758-206a1bd926a3-tls-certs\") pod \"simulator-w1-fd61e3-kserve-649c45966f-hrl97\" (UID: \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\") " pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 18:47:56.336365 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.336331 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-tmp-dir\") pod \"simulator-w1-fd61e3-kserve-649c45966f-hrl97\" (UID: \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\") " pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 18:47:56.336480 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.336339 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-model-cache\") pod \"simulator-w1-fd61e3-kserve-649c45966f-hrl97\" (UID: \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\") " pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 18:47:56.336480 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.336383 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-home\") pod \"simulator-w1-fd61e3-kserve-649c45966f-hrl97\" (UID: \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\") " pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 18:47:56.338524 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.338487 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-dshm\") pod \"simulator-w1-fd61e3-kserve-649c45966f-hrl97\" (UID: \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\") " pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 18:47:56.338632 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.338615 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a5f18bf-96ba-4edf-a758-206a1bd926a3-tls-certs\") pod \"simulator-w1-fd61e3-kserve-649c45966f-hrl97\" (UID: \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\") " pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 18:47:56.398840 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.398810 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65r8x\" (UniqueName: \"kubernetes.io/projected/1a5f18bf-96ba-4edf-a758-206a1bd926a3-kube-api-access-65r8x\") pod \"simulator-w1-fd61e3-kserve-649c45966f-hrl97\" (UID: \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\") " pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 18:47:56.403915 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:47:56.403875 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:47:56.479506 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.479403 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 18:47:56.861945 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:56.861906 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97"] Oct 30 18:47:56.865270 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:47:56.865234 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a5f18bf_96ba_4edf_a758_206a1bd926a3.slice/crio-0818bca23fd7e435589fcfddf57f041692918f435791a954b24b36e37415b271 WatchSource:0}: Error finding container 0818bca23fd7e435589fcfddf57f041692918f435791a954b24b36e37415b271: Status 404 returned error can't find the container with id 0818bca23fd7e435589fcfddf57f041692918f435791a954b24b36e37415b271 Oct 30 18:47:57.798216 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:57.798180 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" event={"ID":"1a5f18bf-96ba-4edf-a758-206a1bd926a3","Type":"ContainerStarted","Data":"9190ef4458cfdbc9586184bedbf49fe9c73e01228e2c297f4ba75f4b0d21fee4"} Oct 30 18:47:57.798660 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:57.798223 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" event={"ID":"1a5f18bf-96ba-4edf-a758-206a1bd926a3","Type":"ContainerStarted","Data":"0818bca23fd7e435589fcfddf57f041692918f435791a954b24b36e37415b271"} Oct 30 18:47:57.915621 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:57.915564 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" podStartSLOduration=1.915548166 podStartE2EDuration="1.915548166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:57.914421528 +0000 UTC m=+837.941526486" watchObservedRunningTime="2026-10-30 18:47:57.915548166 +0000 UTC m=+837.942653125" Oct 30 18:47:58.480248 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:47:58.480203 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 18:48:06.480205 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:06.480157 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 18:48:06.492730 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:06.492690 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 18:48:06.853257 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:06.853229 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 18:48:10.998052 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:10.997980 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z"] Oct 30 18:48:11.001607 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:11.001586 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 18:48:11.079182 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:11.079121 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-model-cache\") pod \"premium-w0-2ade09-kserve-645d756b69-92k4z\" (UID: \"5a868137-eb41-433c-ade6-58eb8c96d623\") " pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 18:48:11.079339 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:11.079188 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k94hr\" (UniqueName: \"kubernetes.io/projected/5a868137-eb41-433c-ade6-58eb8c96d623-kube-api-access-k94hr\") pod \"premium-w0-2ade09-kserve-645d756b69-92k4z\" (UID: \"5a868137-eb41-433c-ade6-58eb8c96d623\") " pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 18:48:11.079339 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:11.079283 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-home\") pod \"premium-w0-2ade09-kserve-645d756b69-92k4z\" (UID: \"5a868137-eb41-433c-ade6-58eb8c96d623\") " pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 18:48:11.079429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:11.079348 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a868137-eb41-433c-ade6-58eb8c96d623-tls-certs\") pod \"premium-w0-2ade09-kserve-645d756b69-92k4z\" (UID: \"5a868137-eb41-433c-ade6-58eb8c96d623\") " pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 18:48:11.079429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:11.079381 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-dshm\") pod \"premium-w0-2ade09-kserve-645d756b69-92k4z\" (UID: \"5a868137-eb41-433c-ade6-58eb8c96d623\") " pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 18:48:11.079429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:11.079402 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-tmp-dir\") pod \"premium-w0-2ade09-kserve-645d756b69-92k4z\" (UID: \"5a868137-eb41-433c-ade6-58eb8c96d623\") " pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 18:48:11.102766 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:11.102737 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2ade09\"/\"premium-w0-2ade09-kserve-self-signed-certs\"" Oct 30 18:48:11.104380 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:11.104352 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z"] Oct 30 18:48:11.179837 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:11.179801 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-model-cache\") pod \"premium-w0-2ade09-kserve-645d756b69-92k4z\" (UID: \"5a868137-eb41-433c-ade6-58eb8c96d623\") " pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 18:48:11.179837 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:11.179841 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k94hr\" (UniqueName: \"kubernetes.io/projected/5a868137-eb41-433c-ade6-58eb8c96d623-kube-api-access-k94hr\") pod \"premium-w0-2ade09-kserve-645d756b69-92k4z\" (UID: \"5a868137-eb41-433c-ade6-58eb8c96d623\") " pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 18:48:11.180088 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:11.179860 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-home\") pod \"premium-w0-2ade09-kserve-645d756b69-92k4z\" (UID: \"5a868137-eb41-433c-ade6-58eb8c96d623\") " pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 18:48:11.180088 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:11.179893 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a868137-eb41-433c-ade6-58eb8c96d623-tls-certs\") pod \"premium-w0-2ade09-kserve-645d756b69-92k4z\" (UID: \"5a868137-eb41-433c-ade6-58eb8c96d623\") " pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 18:48:11.180088 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:11.179929 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-dshm\") pod \"premium-w0-2ade09-kserve-645d756b69-92k4z\" (UID: \"5a868137-eb41-433c-ade6-58eb8c96d623\") " pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 18:48:11.180088 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:11.179952 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-tmp-dir\") pod \"premium-w0-2ade09-kserve-645d756b69-92k4z\" (UID: \"5a868137-eb41-433c-ade6-58eb8c96d623\") " pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 18:48:11.180314 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:11.180264 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-model-cache\") pod \"premium-w0-2ade09-kserve-645d756b69-92k4z\" (UID: \"5a868137-eb41-433c-ade6-58eb8c96d623\") " pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 18:48:11.180396 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:11.180375 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-home\") pod \"premium-w0-2ade09-kserve-645d756b69-92k4z\" (UID: \"5a868137-eb41-433c-ade6-58eb8c96d623\") " pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 18:48:11.180481 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:11.180452 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-tmp-dir\") pod \"premium-w0-2ade09-kserve-645d756b69-92k4z\" (UID: \"5a868137-eb41-433c-ade6-58eb8c96d623\") " pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 18:48:11.182480 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:11.182459 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-dshm\") pod \"premium-w0-2ade09-kserve-645d756b69-92k4z\" (UID: \"5a868137-eb41-433c-ade6-58eb8c96d623\") " pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 18:48:11.182638 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:11.182621 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a868137-eb41-433c-ade6-58eb8c96d623-tls-certs\") pod \"premium-w0-2ade09-kserve-645d756b69-92k4z\" (UID: \"5a868137-eb41-433c-ade6-58eb8c96d623\") " pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 18:48:11.241456 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:11.241413 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k94hr\" (UniqueName: \"kubernetes.io/projected/5a868137-eb41-433c-ade6-58eb8c96d623-kube-api-access-k94hr\") pod \"premium-w0-2ade09-kserve-645d756b69-92k4z\" (UID: \"5a868137-eb41-433c-ade6-58eb8c96d623\") " pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 18:48:11.345763 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:11.345665 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 18:48:11.728983 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:11.728898 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z"] Oct 30 18:48:11.732396 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:48:11.732365 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a868137_eb41_433c_ade6_58eb8c96d623.slice/crio-c29348a1962bddecdda97eee0ad284c4d549182d01d0732aa6fa2d97e3b27854 WatchSource:0}: Error finding container c29348a1962bddecdda97eee0ad284c4d549182d01d0732aa6fa2d97e3b27854: Status 404 returned error can't find the container with id c29348a1962bddecdda97eee0ad284c4d549182d01d0732aa6fa2d97e3b27854 Oct 30 18:48:11.872114 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:11.872074 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" event={"ID":"5a868137-eb41-433c-ade6-58eb8c96d623","Type":"ContainerStarted","Data":"ffaf31cc4895a3a2b0e24725687a341fa9c99a439b7600d812b62278b55b5e9e"} Oct 30 18:48:11.872292 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:11.872120 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" event={"ID":"5a868137-eb41-433c-ade6-58eb8c96d623","Type":"ContainerStarted","Data":"c29348a1962bddecdda97eee0ad284c4d549182d01d0732aa6fa2d97e3b27854"} Oct 30 18:48:11.988249 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:11.988085 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" podStartSLOduration=1.9880689409999999 podStartE2EDuration="1.988068941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:11.987864518 +0000 UTC m=+852.014969491" watchObservedRunningTime="2026-10-30 18:48:11.988068941 +0000 UTC m=+852.015173901" Oct 30 18:48:12.346845 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:12.346808 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 18:48:21.345888 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:21.345844 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 18:48:21.358786 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:21.358749 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 18:48:21.936462 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:48:21.936429 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 18:48:26.397907 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:48:26.397865 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:48:56.396901 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:48:56.396861 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:49:08.295371 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:08.295333 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4844a61-9fzvb"] Oct 30 18:49:08.298925 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:08.298902 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4844a61-9fzvb" Oct 30 18:49:08.397482 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:08.397452 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05a1547-0c46-4b53-b122-b9d7ae9588c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4844a61-9fzvb\" (UID: \"c05a1547-0c46-4b53-b122-b9d7ae9588c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4844a61-9fzvb" Oct 30 18:49:08.397627 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:08.397529 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdt5q\" (UniqueName: \"kubernetes.io/projected/c05a1547-0c46-4b53-b122-b9d7ae9588c4-kube-api-access-gdt5q\") pod \"maas-api-revoke-sub-a4844a61-9fzvb\" (UID: \"c05a1547-0c46-4b53-b122-b9d7ae9588c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4844a61-9fzvb" Oct 30 18:49:08.397627 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:08.397592 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05a1547-0c46-4b53-b122-b9d7ae9588c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4844a61-9fzvb\" (UID: \"c05a1547-0c46-4b53-b122-b9d7ae9588c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4844a61-9fzvb" Oct 30 18:49:08.400199 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:08.400173 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:49:08.400342 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:08.400220 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:49:08.400342 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:08.400242 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:49:08.400342 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:08.400261 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:49:08.402064 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:08.402046 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4844a61-9fzvb"] Oct 30 18:49:08.498348 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:08.498328 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdt5q\" (UniqueName: \"kubernetes.io/projected/c05a1547-0c46-4b53-b122-b9d7ae9588c4-kube-api-access-gdt5q\") pod \"maas-api-revoke-sub-a4844a61-9fzvb\" (UID: \"c05a1547-0c46-4b53-b122-b9d7ae9588c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4844a61-9fzvb" Oct 30 18:49:08.498476 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:08.498354 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05a1547-0c46-4b53-b122-b9d7ae9588c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4844a61-9fzvb\" (UID: \"c05a1547-0c46-4b53-b122-b9d7ae9588c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4844a61-9fzvb" Oct 30 18:49:08.498476 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:08.498413 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05a1547-0c46-4b53-b122-b9d7ae9588c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4844a61-9fzvb\" (UID: \"c05a1547-0c46-4b53-b122-b9d7ae9588c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4844a61-9fzvb" Oct 30 18:49:08.499204 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:08.499119 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05a1547-0c46-4b53-b122-b9d7ae9588c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4844a61-9fzvb\" (UID: \"c05a1547-0c46-4b53-b122-b9d7ae9588c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4844a61-9fzvb" Oct 30 18:49:08.500849 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:08.500828 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05a1547-0c46-4b53-b122-b9d7ae9588c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4844a61-9fzvb\" (UID: \"c05a1547-0c46-4b53-b122-b9d7ae9588c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4844a61-9fzvb" Oct 30 18:49:08.559271 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:08.559213 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdt5q\" (UniqueName: \"kubernetes.io/projected/c05a1547-0c46-4b53-b122-b9d7ae9588c4-kube-api-access-gdt5q\") pod \"maas-api-revoke-sub-a4844a61-9fzvb\" (UID: \"c05a1547-0c46-4b53-b122-b9d7ae9588c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4844a61-9fzvb" Oct 30 18:49:08.611652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:08.611631 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4844a61-9fzvb" Oct 30 18:49:08.986578 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:08.986553 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4844a61-9fzvb"] Oct 30 18:49:08.989259 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:49:08.989232 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc05a1547_0c46_4b53_b122_b9d7ae9588c4.slice/crio-884156fb73da4228d2c6f24255a1c53629baa5a724454a011408a9a2651d7772 WatchSource:0}: Error finding container 884156fb73da4228d2c6f24255a1c53629baa5a724454a011408a9a2651d7772: Status 404 returned error can't find the container with id 884156fb73da4228d2c6f24255a1c53629baa5a724454a011408a9a2651d7772 Oct 30 18:49:09.151921 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:09.151883 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4844a61-9fzvb" event={"ID":"c05a1547-0c46-4b53-b122-b9d7ae9588c4","Type":"ContainerStarted","Data":"884156fb73da4228d2c6f24255a1c53629baa5a724454a011408a9a2651d7772"} Oct 30 18:49:10.158296 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:10.158212 4531 generic.go:356] "Generic (PLEG): container finished" podID="c05a1547-0c46-4b53-b122-b9d7ae9588c4" containerID="b51402c0984199d9255490130dc91c46b9eeae0d5f97bbf437d4a338fd4128ad" exitCode=0 Oct 30 18:49:10.158653 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:10.158290 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4844a61-9fzvb" event={"ID":"c05a1547-0c46-4b53-b122-b9d7ae9588c4","Type":"ContainerDied","Data":"b51402c0984199d9255490130dc91c46b9eeae0d5f97bbf437d4a338fd4128ad"} Oct 30 18:49:10.526402 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:10.526370 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8"] Oct 30 18:49:10.529695 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:10.529680 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 18:49:10.582468 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:10.582444 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fd61e3\"/\"premium-w1-fd61e3-kserve-self-signed-certs\"" Oct 30 18:49:10.591317 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:10.591292 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8"] Oct 30 18:49:10.618068 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:10.618043 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-tmp-dir\") pod \"premium-w1-fd61e3-kserve-584cf676-dbzx8\" (UID: \"0679a60b-f854-49dd-a87d-3ea640cac995\") " pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 18:49:10.618232 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:10.618075 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-home\") pod \"premium-w1-fd61e3-kserve-584cf676-dbzx8\" (UID: \"0679a60b-f854-49dd-a87d-3ea640cac995\") " pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 18:49:10.618232 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:10.618101 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0679a60b-f854-49dd-a87d-3ea640cac995-tls-certs\") pod \"premium-w1-fd61e3-kserve-584cf676-dbzx8\" (UID: \"0679a60b-f854-49dd-a87d-3ea640cac995\") " pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 18:49:10.618352 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:10.618263 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-model-cache\") pod \"premium-w1-fd61e3-kserve-584cf676-dbzx8\" (UID: \"0679a60b-f854-49dd-a87d-3ea640cac995\") " pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 18:49:10.618352 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:10.618303 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pf9l\" (UniqueName: \"kubernetes.io/projected/0679a60b-f854-49dd-a87d-3ea640cac995-kube-api-access-5pf9l\") pod \"premium-w1-fd61e3-kserve-584cf676-dbzx8\" (UID: \"0679a60b-f854-49dd-a87d-3ea640cac995\") " pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 18:49:10.618448 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:10.618430 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-dshm\") pod \"premium-w1-fd61e3-kserve-584cf676-dbzx8\" (UID: \"0679a60b-f854-49dd-a87d-3ea640cac995\") " pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 18:49:10.719823 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:10.719788 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-dshm\") pod \"premium-w1-fd61e3-kserve-584cf676-dbzx8\" (UID: \"0679a60b-f854-49dd-a87d-3ea640cac995\") " pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 18:49:10.720001 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:10.719857 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-tmp-dir\") pod \"premium-w1-fd61e3-kserve-584cf676-dbzx8\" (UID: \"0679a60b-f854-49dd-a87d-3ea640cac995\") " pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 18:49:10.720001 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:10.719882 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-home\") pod \"premium-w1-fd61e3-kserve-584cf676-dbzx8\" (UID: \"0679a60b-f854-49dd-a87d-3ea640cac995\") " pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 18:49:10.720001 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:10.719921 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0679a60b-f854-49dd-a87d-3ea640cac995-tls-certs\") pod \"premium-w1-fd61e3-kserve-584cf676-dbzx8\" (UID: \"0679a60b-f854-49dd-a87d-3ea640cac995\") " pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 18:49:10.720001 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:10.719984 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-model-cache\") pod \"premium-w1-fd61e3-kserve-584cf676-dbzx8\" (UID: \"0679a60b-f854-49dd-a87d-3ea640cac995\") " pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 18:49:10.720241 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:10.720012 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pf9l\" (UniqueName: \"kubernetes.io/projected/0679a60b-f854-49dd-a87d-3ea640cac995-kube-api-access-5pf9l\") pod \"premium-w1-fd61e3-kserve-584cf676-dbzx8\" (UID: \"0679a60b-f854-49dd-a87d-3ea640cac995\") " pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 18:49:10.720346 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:10.720321 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-tmp-dir\") pod \"premium-w1-fd61e3-kserve-584cf676-dbzx8\" (UID: \"0679a60b-f854-49dd-a87d-3ea640cac995\") " pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 18:49:10.720440 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:10.720381 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-home\") pod \"premium-w1-fd61e3-kserve-584cf676-dbzx8\" (UID: \"0679a60b-f854-49dd-a87d-3ea640cac995\") " pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 18:49:10.720697 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:10.720669 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-model-cache\") pod \"premium-w1-fd61e3-kserve-584cf676-dbzx8\" (UID: \"0679a60b-f854-49dd-a87d-3ea640cac995\") " pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 18:49:10.722211 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:10.722190 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-dshm\") pod \"premium-w1-fd61e3-kserve-584cf676-dbzx8\" (UID: \"0679a60b-f854-49dd-a87d-3ea640cac995\") " pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 18:49:10.722808 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:10.722790 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0679a60b-f854-49dd-a87d-3ea640cac995-tls-certs\") pod \"premium-w1-fd61e3-kserve-584cf676-dbzx8\" (UID: \"0679a60b-f854-49dd-a87d-3ea640cac995\") " pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 18:49:10.813632 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:10.813562 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pf9l\" (UniqueName: \"kubernetes.io/projected/0679a60b-f854-49dd-a87d-3ea640cac995-kube-api-access-5pf9l\") pod \"premium-w1-fd61e3-kserve-584cf676-dbzx8\" (UID: \"0679a60b-f854-49dd-a87d-3ea640cac995\") " pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 18:49:10.842800 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:10.842776 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 18:49:11.232063 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:11.232025 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8"] Oct 30 18:49:11.234808 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:49:11.234780 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0679a60b_f854_49dd_a87d_3ea640cac995.slice/crio-df901ee41d32b1faa7c89f470b36bb46478769f74b24223a095ec322656d0520 WatchSource:0}: Error finding container df901ee41d32b1faa7c89f470b36bb46478769f74b24223a095ec322656d0520: Status 404 returned error can't find the container with id df901ee41d32b1faa7c89f470b36bb46478769f74b24223a095ec322656d0520 Oct 30 18:49:11.301906 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:11.301882 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4844a61-9fzvb" Oct 30 18:49:11.326698 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:11.326672 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c05a1547-0c46-4b53-b122-b9d7ae9588c4-kube-api-access-gdt5q\" (UniqueName: \"kubernetes.io/projected/c05a1547-0c46-4b53-b122-b9d7ae9588c4-kube-api-access-gdt5q\") pod \"c05a1547-0c46-4b53-b122-b9d7ae9588c4\" (UID: \"c05a1547-0c46-4b53-b122-b9d7ae9588c4\") " Oct 30 18:49:11.326806 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:11.326738 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c05a1547-0c46-4b53-b122-b9d7ae9588c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05a1547-0c46-4b53-b122-b9d7ae9588c4-maas-api-openshift-ca\") pod \"c05a1547-0c46-4b53-b122-b9d7ae9588c4\" (UID: \"c05a1547-0c46-4b53-b122-b9d7ae9588c4\") " Oct 30 18:49:11.326806 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:11.326767 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c05a1547-0c46-4b53-b122-b9d7ae9588c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05a1547-0c46-4b53-b122-b9d7ae9588c4-maas-api-xks-ca\") pod \"c05a1547-0c46-4b53-b122-b9d7ae9588c4\" (UID: \"c05a1547-0c46-4b53-b122-b9d7ae9588c4\") " Oct 30 18:49:11.327316 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:11.327281 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c05a1547-0c46-4b53-b122-b9d7ae9588c4-maas-api-openshift-ca" pod "c05a1547-0c46-4b53-b122-b9d7ae9588c4" (UID: "c05a1547-0c46-4b53-b122-b9d7ae9588c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:49:11.329175 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:11.329097 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c05a1547-0c46-4b53-b122-b9d7ae9588c4-maas-api-xks-ca" pod "c05a1547-0c46-4b53-b122-b9d7ae9588c4" (UID: "c05a1547-0c46-4b53-b122-b9d7ae9588c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:49:11.329257 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:11.329216 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c05a1547-0c46-4b53-b122-b9d7ae9588c4-kube-api-access-gdt5q" pod "c05a1547-0c46-4b53-b122-b9d7ae9588c4" (UID: "c05a1547-0c46-4b53-b122-b9d7ae9588c4"). InnerVolumeSpecName "kube-api-access-gdt5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:49:11.428261 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:11.428200 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdt5q\" (UniqueName: \"kubernetes.io/projected/c05a1547-0c46-4b53-b122-b9d7ae9588c4-kube-api-access-gdt5q\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:49:11.428261 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:11.428225 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05a1547-0c46-4b53-b122-b9d7ae9588c4-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:49:11.428261 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:11.428237 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05a1547-0c46-4b53-b122-b9d7ae9588c4-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:49:12.169481 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:12.169445 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" event={"ID":"0679a60b-f854-49dd-a87d-3ea640cac995","Type":"ContainerStarted","Data":"cc78196bd0fb9c1faa36e9bbf14fe4cde15e8a49b6b1fd45406222a53368427a"} Oct 30 18:49:12.169622 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:12.169496 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" event={"ID":"0679a60b-f854-49dd-a87d-3ea640cac995","Type":"ContainerStarted","Data":"df901ee41d32b1faa7c89f470b36bb46478769f74b24223a095ec322656d0520"} Oct 30 18:49:12.170820 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:12.170796 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4844a61-9fzvb" Oct 30 18:49:12.170820 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:12.170812 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4844a61-9fzvb" event={"ID":"c05a1547-0c46-4b53-b122-b9d7ae9588c4","Type":"ContainerDied","Data":"884156fb73da4228d2c6f24255a1c53629baa5a724454a011408a9a2651d7772"} Oct 30 18:49:12.170984 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:12.170848 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="884156fb73da4228d2c6f24255a1c53629baa5a724454a011408a9a2651d7772" Oct 30 18:49:12.286612 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:12.286564 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" podStartSLOduration=2.286549816 podStartE2EDuration="2.286549816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:12.286326927 +0000 UTC m=+912.313431888" watchObservedRunningTime="2026-10-30 18:49:12.286549816 +0000 UTC m=+912.313654779" Oct 30 18:49:12.843399 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:12.843368 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 18:49:19.565670 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:19.565642 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64384eda-fdfks"] Oct 30 18:49:19.566057 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:19.566043 4531 cpu_manager.go:414] "containerMap: removing container" podUID="c05a1547-0c46-4b53-b122-b9d7ae9588c4" containerName="revoke-keys" Oct 30 18:49:19.566099 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:19.566058 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c05a1547-0c46-4b53-b122-b9d7ae9588c4" containerName="revoke-keys" Oct 30 18:49:19.566168 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:19.566157 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c05a1547-0c46-4b53-b122-b9d7ae9588c4" containerName="revoke-keys" Oct 30 18:49:19.572061 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:19.572045 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64384eda-fdfks" Oct 30 18:49:19.627099 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:19.627071 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:49:19.627233 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:19.627144 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:49:19.676301 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:19.676279 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:49:19.676681 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:19.676658 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:49:19.677885 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:19.677867 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64384eda-fdfks"] Oct 30 18:49:19.697065 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:19.697041 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64384eda-fdfks\" (UID: \"4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64384eda-fdfks" Oct 30 18:49:19.697162 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:19.697077 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64384eda-fdfks\" (UID: \"4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64384eda-fdfks" Oct 30 18:49:19.697219 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:19.697196 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f74pd\" (UniqueName: \"kubernetes.io/projected/4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc-kube-api-access-f74pd\") pod \"maas-api-revoke-sub-64384eda-fdfks\" (UID: \"4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64384eda-fdfks" Oct 30 18:49:19.798215 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:19.798187 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64384eda-fdfks\" (UID: \"4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64384eda-fdfks" Oct 30 18:49:19.798343 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:19.798225 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64384eda-fdfks\" (UID: \"4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64384eda-fdfks" Oct 30 18:49:19.798343 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:19.798267 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f74pd\" (UniqueName: \"kubernetes.io/projected/4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc-kube-api-access-f74pd\") pod \"maas-api-revoke-sub-64384eda-fdfks\" (UID: \"4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64384eda-fdfks" Oct 30 18:49:19.798930 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:19.798903 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64384eda-fdfks\" (UID: \"4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64384eda-fdfks" Oct 30 18:49:19.800674 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:19.800655 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64384eda-fdfks\" (UID: \"4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64384eda-fdfks" Oct 30 18:49:19.859376 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:19.859321 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f74pd\" (UniqueName: \"kubernetes.io/projected/4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc-kube-api-access-f74pd\") pod \"maas-api-revoke-sub-64384eda-fdfks\" (UID: \"4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64384eda-fdfks" Oct 30 18:49:19.884300 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:19.884274 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64384eda-fdfks" Oct 30 18:49:20.257011 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:20.256929 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64384eda-fdfks"] Oct 30 18:49:20.259220 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:49:20.259195 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b9f9bfc_a3e1_4c34_9894_4ae0632b90cc.slice/crio-c61ec1e4f8dd57a623debe121d951597f48465b76f516a0a3e3e696f4d0fd082 WatchSource:0}: Error finding container c61ec1e4f8dd57a623debe121d951597f48465b76f516a0a3e3e696f4d0fd082: Status 404 returned error can't find the container with id c61ec1e4f8dd57a623debe121d951597f48465b76f516a0a3e3e696f4d0fd082 Oct 30 18:49:20.548177 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:20.548149 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e8bc64-47w74"] Oct 30 18:49:20.551762 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:20.551746 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e8bc64-47w74" Oct 30 18:49:20.607272 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:20.607245 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d71500-74ce-4e8a-a514-875df485349b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48e8bc64-47w74\" (UID: \"95d71500-74ce-4e8a-a514-875df485349b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e8bc64-47w74" Oct 30 18:49:20.607637 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:20.607280 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5p5c\" (UniqueName: \"kubernetes.io/projected/95d71500-74ce-4e8a-a514-875df485349b-kube-api-access-t5p5c\") pod \"maas-api-revoke-sub-48e8bc64-47w74\" (UID: \"95d71500-74ce-4e8a-a514-875df485349b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e8bc64-47w74" Oct 30 18:49:20.607637 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:20.607402 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d71500-74ce-4e8a-a514-875df485349b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48e8bc64-47w74\" (UID: \"95d71500-74ce-4e8a-a514-875df485349b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e8bc64-47w74" Oct 30 18:49:20.612218 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:20.612196 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e8bc64-47w74"] Oct 30 18:49:20.708283 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:20.708260 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d71500-74ce-4e8a-a514-875df485349b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48e8bc64-47w74\" (UID: \"95d71500-74ce-4e8a-a514-875df485349b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e8bc64-47w74" Oct 30 18:49:20.708411 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:20.708287 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5p5c\" (UniqueName: \"kubernetes.io/projected/95d71500-74ce-4e8a-a514-875df485349b-kube-api-access-t5p5c\") pod \"maas-api-revoke-sub-48e8bc64-47w74\" (UID: \"95d71500-74ce-4e8a-a514-875df485349b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e8bc64-47w74" Oct 30 18:49:20.708411 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:20.708313 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d71500-74ce-4e8a-a514-875df485349b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48e8bc64-47w74\" (UID: \"95d71500-74ce-4e8a-a514-875df485349b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e8bc64-47w74" Oct 30 18:49:20.708939 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:20.708910 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d71500-74ce-4e8a-a514-875df485349b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48e8bc64-47w74\" (UID: \"95d71500-74ce-4e8a-a514-875df485349b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e8bc64-47w74" Oct 30 18:49:20.710733 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:20.710708 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d71500-74ce-4e8a-a514-875df485349b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48e8bc64-47w74\" (UID: \"95d71500-74ce-4e8a-a514-875df485349b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e8bc64-47w74" Oct 30 18:49:20.769614 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:20.769586 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5p5c\" (UniqueName: \"kubernetes.io/projected/95d71500-74ce-4e8a-a514-875df485349b-kube-api-access-t5p5c\") pod \"maas-api-revoke-sub-48e8bc64-47w74\" (UID: \"95d71500-74ce-4e8a-a514-875df485349b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e8bc64-47w74" Oct 30 18:49:20.842949 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:20.842887 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 18:49:20.855431 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:20.855413 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 18:49:20.863675 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:20.863649 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e8bc64-47w74" Oct 30 18:49:21.212763 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:21.212736 4531 generic.go:356] "Generic (PLEG): container finished" podID="4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc" containerID="062c70f66bed8015f48450f6839ab1543ff882a566ecf83322f4fb933122693f" exitCode=0 Oct 30 18:49:21.212913 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:21.212767 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64384eda-fdfks" event={"ID":"4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc","Type":"ContainerDied","Data":"062c70f66bed8015f48450f6839ab1543ff882a566ecf83322f4fb933122693f"} Oct 30 18:49:21.212913 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:21.212819 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64384eda-fdfks" event={"ID":"4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc","Type":"ContainerStarted","Data":"c61ec1e4f8dd57a623debe121d951597f48465b76f516a0a3e3e696f4d0fd082"} Oct 30 18:49:21.224635 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:21.224617 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 18:49:21.243694 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:21.243675 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e8bc64-47w74"] Oct 30 18:49:21.246641 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:49:21.246615 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d71500_74ce_4e8a_a514_875df485349b.slice/crio-682e6f65641af3d2d3782fda1991c06595f86e38b0432eac0521b24321e240e9 WatchSource:0}: Error finding container 682e6f65641af3d2d3782fda1991c06595f86e38b0432eac0521b24321e240e9: Status 404 returned error can't find the container with id 682e6f65641af3d2d3782fda1991c06595f86e38b0432eac0521b24321e240e9 Oct 30 18:49:22.219340 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:22.219309 4531 generic.go:356] "Generic (PLEG): container finished" podID="95d71500-74ce-4e8a-a514-875df485349b" containerID="cc44cb9a84286dba1efa1680d8f9fa22f22b2a9fb5b981e36694658b2edd25db" exitCode=0 Oct 30 18:49:22.219746 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:22.219350 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e8bc64-47w74" event={"ID":"95d71500-74ce-4e8a-a514-875df485349b","Type":"ContainerDied","Data":"cc44cb9a84286dba1efa1680d8f9fa22f22b2a9fb5b981e36694658b2edd25db"} Oct 30 18:49:22.219746 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:22.219408 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e8bc64-47w74" event={"ID":"95d71500-74ce-4e8a-a514-875df485349b","Type":"ContainerStarted","Data":"682e6f65641af3d2d3782fda1991c06595f86e38b0432eac0521b24321e240e9"} Oct 30 18:49:22.346583 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:22.346562 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64384eda-fdfks" Oct 30 18:49:22.422879 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:22.422855 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc-maas-api-openshift-ca\") pod \"4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc\" (UID: \"4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc\") " Oct 30 18:49:22.423046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:22.422925 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc-kube-api-access-f74pd\" (UniqueName: \"kubernetes.io/projected/4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc-kube-api-access-f74pd\") pod \"4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc\" (UID: \"4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc\") " Oct 30 18:49:22.423046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:22.423030 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc-maas-api-xks-ca\") pod \"4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc\" (UID: \"4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc\") " Oct 30 18:49:22.423284 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:22.423261 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc-maas-api-openshift-ca" pod "4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc" (UID: "4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:49:22.423363 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:22.423331 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:49:22.425176 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:22.425152 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc-kube-api-access-f74pd" pod "4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc" (UID: "4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc"). InnerVolumeSpecName "kube-api-access-f74pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:49:22.425259 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:22.425192 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc-maas-api-xks-ca" pod "4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc" (UID: "4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:49:22.524076 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:22.524050 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f74pd\" (UniqueName: \"kubernetes.io/projected/4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc-kube-api-access-f74pd\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:49:22.524076 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:22.524073 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:49:23.227101 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:23.227066 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64384eda-fdfks" Oct 30 18:49:23.227523 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:23.227357 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64384eda-fdfks" event={"ID":"4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc","Type":"ContainerDied","Data":"c61ec1e4f8dd57a623debe121d951597f48465b76f516a0a3e3e696f4d0fd082"} Oct 30 18:49:23.227523 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:23.227423 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c61ec1e4f8dd57a623debe121d951597f48465b76f516a0a3e3e696f4d0fd082" Oct 30 18:49:23.357428 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:23.357400 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e8bc64-47w74" Oct 30 18:49:23.430765 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:23.430739 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95d71500-74ce-4e8a-a514-875df485349b-kube-api-access-t5p5c\" (UniqueName: \"kubernetes.io/projected/95d71500-74ce-4e8a-a514-875df485349b-kube-api-access-t5p5c\") pod \"95d71500-74ce-4e8a-a514-875df485349b\" (UID: \"95d71500-74ce-4e8a-a514-875df485349b\") " Oct 30 18:49:23.430875 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:23.430795 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95d71500-74ce-4e8a-a514-875df485349b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d71500-74ce-4e8a-a514-875df485349b-maas-api-openshift-ca\") pod \"95d71500-74ce-4e8a-a514-875df485349b\" (UID: \"95d71500-74ce-4e8a-a514-875df485349b\") " Oct 30 18:49:23.430928 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:23.430883 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95d71500-74ce-4e8a-a514-875df485349b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d71500-74ce-4e8a-a514-875df485349b-maas-api-xks-ca\") pod \"95d71500-74ce-4e8a-a514-875df485349b\" (UID: \"95d71500-74ce-4e8a-a514-875df485349b\") " Oct 30 18:49:23.431187 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:23.431162 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95d71500-74ce-4e8a-a514-875df485349b-maas-api-openshift-ca" pod "95d71500-74ce-4e8a-a514-875df485349b" (UID: "95d71500-74ce-4e8a-a514-875df485349b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:49:23.432949 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:23.432927 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95d71500-74ce-4e8a-a514-875df485349b-maas-api-xks-ca" pod "95d71500-74ce-4e8a-a514-875df485349b" (UID: "95d71500-74ce-4e8a-a514-875df485349b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:49:23.433075 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:23.433058 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d71500-74ce-4e8a-a514-875df485349b-kube-api-access-t5p5c" pod "95d71500-74ce-4e8a-a514-875df485349b" (UID: "95d71500-74ce-4e8a-a514-875df485349b"). InnerVolumeSpecName "kube-api-access-t5p5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:49:23.532038 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:23.532011 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d71500-74ce-4e8a-a514-875df485349b-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:49:23.532038 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:23.532038 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5p5c\" (UniqueName: \"kubernetes.io/projected/95d71500-74ce-4e8a-a514-875df485349b-kube-api-access-t5p5c\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:49:23.532232 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:23.532049 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d71500-74ce-4e8a-a514-875df485349b-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:49:24.232292 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:24.232260 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e8bc64-47w74" Oct 30 18:49:24.232645 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:24.232302 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e8bc64-47w74" event={"ID":"95d71500-74ce-4e8a-a514-875df485349b","Type":"ContainerDied","Data":"682e6f65641af3d2d3782fda1991c06595f86e38b0432eac0521b24321e240e9"} Oct 30 18:49:24.232645 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:24.232351 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="682e6f65641af3d2d3782fda1991c06595f86e38b0432eac0521b24321e240e9" Oct 30 18:49:26.129314 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:26.129287 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0900ec82-pb4jr"] Oct 30 18:49:26.129783 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:26.129663 4531 cpu_manager.go:414] "containerMap: removing container" podUID="4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc" containerName="revoke-keys" Oct 30 18:49:26.129783 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:26.129678 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc" containerName="revoke-keys" Oct 30 18:49:26.129783 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:26.129687 4531 cpu_manager.go:414] "containerMap: removing container" podUID="95d71500-74ce-4e8a-a514-875df485349b" containerName="revoke-keys" Oct 30 18:49:26.129783 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:26.129692 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95d71500-74ce-4e8a-a514-875df485349b" containerName="revoke-keys" Oct 30 18:49:26.129930 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:26.129789 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc" containerName="revoke-keys" Oct 30 18:49:26.129930 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:26.129797 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="95d71500-74ce-4e8a-a514-875df485349b" containerName="revoke-keys" Oct 30 18:49:26.133013 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:26.132999 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0900ec82-pb4jr" Oct 30 18:49:26.185261 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:26.185238 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:49:26.234638 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:26.234606 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:49:26.234638 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:26.234615 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:49:26.234638 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:26.234633 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:49:26.236420 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:26.236401 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0900ec82-pb4jr"] Oct 30 18:49:26.252354 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:26.252324 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b7f47d2-b54d-4906-9597-5fe9188cd5c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0900ec82-pb4jr\" (UID: \"2b7f47d2-b54d-4906-9597-5fe9188cd5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0900ec82-pb4jr" Oct 30 18:49:26.252481 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:26.252462 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2rcx\" (UniqueName: \"kubernetes.io/projected/2b7f47d2-b54d-4906-9597-5fe9188cd5c3-kube-api-access-t2rcx\") pod \"maas-api-revoke-sub-0900ec82-pb4jr\" (UID: \"2b7f47d2-b54d-4906-9597-5fe9188cd5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0900ec82-pb4jr" Oct 30 18:49:26.252539 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:26.252528 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b7f47d2-b54d-4906-9597-5fe9188cd5c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0900ec82-pb4jr\" (UID: \"2b7f47d2-b54d-4906-9597-5fe9188cd5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0900ec82-pb4jr" Oct 30 18:49:26.353863 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:26.353834 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b7f47d2-b54d-4906-9597-5fe9188cd5c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0900ec82-pb4jr\" (UID: \"2b7f47d2-b54d-4906-9597-5fe9188cd5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0900ec82-pb4jr" Oct 30 18:49:26.354014 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:26.353869 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b7f47d2-b54d-4906-9597-5fe9188cd5c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0900ec82-pb4jr\" (UID: \"2b7f47d2-b54d-4906-9597-5fe9188cd5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0900ec82-pb4jr" Oct 30 18:49:26.354014 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:26.353919 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2rcx\" (UniqueName: \"kubernetes.io/projected/2b7f47d2-b54d-4906-9597-5fe9188cd5c3-kube-api-access-t2rcx\") pod \"maas-api-revoke-sub-0900ec82-pb4jr\" (UID: \"2b7f47d2-b54d-4906-9597-5fe9188cd5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0900ec82-pb4jr" Oct 30 18:49:26.354544 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:26.354518 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b7f47d2-b54d-4906-9597-5fe9188cd5c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0900ec82-pb4jr\" (UID: \"2b7f47d2-b54d-4906-9597-5fe9188cd5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0900ec82-pb4jr" Oct 30 18:49:26.356416 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:26.356398 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b7f47d2-b54d-4906-9597-5fe9188cd5c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0900ec82-pb4jr\" (UID: \"2b7f47d2-b54d-4906-9597-5fe9188cd5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0900ec82-pb4jr" Oct 30 18:49:26.396454 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:49:26.396392 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:49:26.414912 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:26.414887 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2rcx\" (UniqueName: \"kubernetes.io/projected/2b7f47d2-b54d-4906-9597-5fe9188cd5c3-kube-api-access-t2rcx\") pod \"maas-api-revoke-sub-0900ec82-pb4jr\" (UID: \"2b7f47d2-b54d-4906-9597-5fe9188cd5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0900ec82-pb4jr" Oct 30 18:49:26.444851 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:26.444830 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0900ec82-pb4jr" Oct 30 18:49:26.818482 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:26.818462 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0900ec82-pb4jr"] Oct 30 18:49:26.820868 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:49:26.820839 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7f47d2_b54d_4906_9597_5fe9188cd5c3.slice/crio-fabc57c8bde92e350c18fb2c52e7bc74424a0e9ffd50e1f3097e8e5dbb4d7a0b WatchSource:0}: Error finding container fabc57c8bde92e350c18fb2c52e7bc74424a0e9ffd50e1f3097e8e5dbb4d7a0b: Status 404 returned error can't find the container with id fabc57c8bde92e350c18fb2c52e7bc74424a0e9ffd50e1f3097e8e5dbb4d7a0b Oct 30 18:49:27.247083 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:27.247039 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0900ec82-pb4jr" event={"ID":"2b7f47d2-b54d-4906-9597-5fe9188cd5c3","Type":"ContainerStarted","Data":"47cac700bdb5990f119e2dc3a0feb16e63189bf81c974737694f00f2f29ff232"} Oct 30 18:49:27.247431 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:27.247099 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0900ec82-pb4jr" event={"ID":"2b7f47d2-b54d-4906-9597-5fe9188cd5c3","Type":"ContainerStarted","Data":"fabc57c8bde92e350c18fb2c52e7bc74424a0e9ffd50e1f3097e8e5dbb4d7a0b"} Oct 30 18:49:28.253373 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:28.253327 4531 generic.go:356] "Generic (PLEG): container finished" podID="2b7f47d2-b54d-4906-9597-5fe9188cd5c3" containerID="47cac700bdb5990f119e2dc3a0feb16e63189bf81c974737694f00f2f29ff232" exitCode=0 Oct 30 18:49:28.253806 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:28.253408 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0900ec82-pb4jr" event={"ID":"2b7f47d2-b54d-4906-9597-5fe9188cd5c3","Type":"ContainerDied","Data":"47cac700bdb5990f119e2dc3a0feb16e63189bf81c974737694f00f2f29ff232"} Oct 30 18:49:29.385454 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:29.385430 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0900ec82-pb4jr" Oct 30 18:49:29.482918 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:29.482889 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b7f47d2-b54d-4906-9597-5fe9188cd5c3-kube-api-access-t2rcx\" (UniqueName: \"kubernetes.io/projected/2b7f47d2-b54d-4906-9597-5fe9188cd5c3-kube-api-access-t2rcx\") pod \"2b7f47d2-b54d-4906-9597-5fe9188cd5c3\" (UID: \"2b7f47d2-b54d-4906-9597-5fe9188cd5c3\") " Oct 30 18:49:29.483056 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:29.482925 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b7f47d2-b54d-4906-9597-5fe9188cd5c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b7f47d2-b54d-4906-9597-5fe9188cd5c3-maas-api-xks-ca\") pod \"2b7f47d2-b54d-4906-9597-5fe9188cd5c3\" (UID: \"2b7f47d2-b54d-4906-9597-5fe9188cd5c3\") " Oct 30 18:49:29.483056 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:29.482975 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b7f47d2-b54d-4906-9597-5fe9188cd5c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b7f47d2-b54d-4906-9597-5fe9188cd5c3-maas-api-openshift-ca\") pod \"2b7f47d2-b54d-4906-9597-5fe9188cd5c3\" (UID: \"2b7f47d2-b54d-4906-9597-5fe9188cd5c3\") " Oct 30 18:49:29.483414 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:29.483384 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b7f47d2-b54d-4906-9597-5fe9188cd5c3-maas-api-openshift-ca" pod "2b7f47d2-b54d-4906-9597-5fe9188cd5c3" (UID: "2b7f47d2-b54d-4906-9597-5fe9188cd5c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:49:29.485112 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:29.485089 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b7f47d2-b54d-4906-9597-5fe9188cd5c3-maas-api-xks-ca" pod "2b7f47d2-b54d-4906-9597-5fe9188cd5c3" (UID: "2b7f47d2-b54d-4906-9597-5fe9188cd5c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:49:29.485254 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:29.485235 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b7f47d2-b54d-4906-9597-5fe9188cd5c3-kube-api-access-t2rcx" pod "2b7f47d2-b54d-4906-9597-5fe9188cd5c3" (UID: "2b7f47d2-b54d-4906-9597-5fe9188cd5c3"). InnerVolumeSpecName "kube-api-access-t2rcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:49:29.584635 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:29.584555 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2rcx\" (UniqueName: \"kubernetes.io/projected/2b7f47d2-b54d-4906-9597-5fe9188cd5c3-kube-api-access-t2rcx\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:49:29.584635 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:29.584582 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b7f47d2-b54d-4906-9597-5fe9188cd5c3-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:49:29.584635 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:29.584591 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b7f47d2-b54d-4906-9597-5fe9188cd5c3-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:49:30.264277 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:30.264237 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0900ec82-pb4jr" event={"ID":"2b7f47d2-b54d-4906-9597-5fe9188cd5c3","Type":"ContainerDied","Data":"fabc57c8bde92e350c18fb2c52e7bc74424a0e9ffd50e1f3097e8e5dbb4d7a0b"} Oct 30 18:49:30.264419 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:30.264284 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fabc57c8bde92e350c18fb2c52e7bc74424a0e9ffd50e1f3097e8e5dbb4d7a0b" Oct 30 18:49:30.264419 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:30.264245 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0900ec82-pb4jr" Oct 30 18:49:35.078273 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:35.078239 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce96a591-4dcp6"] Oct 30 18:49:35.078706 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:35.078671 4531 cpu_manager.go:414] "containerMap: removing container" podUID="2b7f47d2-b54d-4906-9597-5fe9188cd5c3" containerName="revoke-keys" Oct 30 18:49:35.078706 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:35.078686 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b7f47d2-b54d-4906-9597-5fe9188cd5c3" containerName="revoke-keys" Oct 30 18:49:35.078786 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:35.078772 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b7f47d2-b54d-4906-9597-5fe9188cd5c3" containerName="revoke-keys" Oct 30 18:49:35.081824 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:35.081809 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce96a591-4dcp6" Oct 30 18:49:35.183492 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:35.183462 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:49:35.183492 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:35.183482 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:49:35.183492 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:35.183463 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:49:35.183492 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:35.183463 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:49:35.185499 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:35.185480 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce96a591-4dcp6"] Oct 30 18:49:35.232382 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:35.232351 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9ql7\" (UniqueName: \"kubernetes.io/projected/fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c-kube-api-access-l9ql7\") pod \"maas-api-revoke-sub-ce96a591-4dcp6\" (UID: \"fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce96a591-4dcp6" Oct 30 18:49:35.232500 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:35.232392 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce96a591-4dcp6\" (UID: \"fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce96a591-4dcp6" Oct 30 18:49:35.232500 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:35.232472 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce96a591-4dcp6\" (UID: \"fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce96a591-4dcp6" Oct 30 18:49:35.333122 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:35.333062 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce96a591-4dcp6\" (UID: \"fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce96a591-4dcp6" Oct 30 18:49:35.333122 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:35.333104 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce96a591-4dcp6\" (UID: \"fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce96a591-4dcp6" Oct 30 18:49:35.333292 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:35.333192 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9ql7\" (UniqueName: \"kubernetes.io/projected/fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c-kube-api-access-l9ql7\") pod \"maas-api-revoke-sub-ce96a591-4dcp6\" (UID: \"fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce96a591-4dcp6" Oct 30 18:49:35.333715 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:35.333694 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce96a591-4dcp6\" (UID: \"fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce96a591-4dcp6" Oct 30 18:49:35.335548 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:35.335527 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce96a591-4dcp6\" (UID: \"fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce96a591-4dcp6" Oct 30 18:49:35.393850 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:35.393823 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9ql7\" (UniqueName: \"kubernetes.io/projected/fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c-kube-api-access-l9ql7\") pod \"maas-api-revoke-sub-ce96a591-4dcp6\" (UID: \"fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce96a591-4dcp6" Oct 30 18:49:35.693911 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:35.693834 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce96a591-4dcp6" Oct 30 18:49:36.064415 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:36.064383 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce96a591-4dcp6"] Oct 30 18:49:36.067864 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:49:36.067836 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe1abf4c_f6c1_4b5d_a16b_ea1544d3fa9c.slice/crio-cbb9a3b5c54bd24e1cbe5152c1d020e38bd510e508a8487a868b504868b04cfc WatchSource:0}: Error finding container cbb9a3b5c54bd24e1cbe5152c1d020e38bd510e508a8487a868b504868b04cfc: Status 404 returned error can't find the container with id cbb9a3b5c54bd24e1cbe5152c1d020e38bd510e508a8487a868b504868b04cfc Oct 30 18:49:36.294537 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:36.294498 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce96a591-4dcp6" event={"ID":"fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c","Type":"ContainerStarted","Data":"cbb9a3b5c54bd24e1cbe5152c1d020e38bd510e508a8487a868b504868b04cfc"} Oct 30 18:49:37.304016 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:37.303986 4531 generic.go:356] "Generic (PLEG): container finished" podID="fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c" containerID="cf5456e5a640bf5d601e963edda4c401744e1495b0363219a00ee543dfc3c007" exitCode=0 Oct 30 18:49:37.304389 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:37.304074 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce96a591-4dcp6" event={"ID":"fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c","Type":"ContainerDied","Data":"cf5456e5a640bf5d601e963edda4c401744e1495b0363219a00ee543dfc3c007"} Oct 30 18:49:38.435397 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:38.435374 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce96a591-4dcp6" Oct 30 18:49:38.563905 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:38.563825 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c-maas-api-openshift-ca\") pod \"fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c\" (UID: \"fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c\") " Oct 30 18:49:38.563905 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:38.563873 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c-maas-api-xks-ca\") pod \"fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c\" (UID: \"fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c\") " Oct 30 18:49:38.564121 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:38.563997 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c-kube-api-access-l9ql7\" (UniqueName: \"kubernetes.io/projected/fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c-kube-api-access-l9ql7\") pod \"fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c\" (UID: \"fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c\") " Oct 30 18:49:38.564243 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:38.564222 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c-maas-api-openshift-ca" pod "fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c" (UID: "fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:49:38.564364 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:38.564345 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:49:38.566077 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:38.566043 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c-maas-api-xks-ca" pod "fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c" (UID: "fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:49:38.566200 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:38.566072 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c-kube-api-access-l9ql7" pod "fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c" (UID: "fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c"). InnerVolumeSpecName "kube-api-access-l9ql7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:49:38.665225 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:38.665200 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:49:38.665225 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:38.665222 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9ql7\" (UniqueName: \"kubernetes.io/projected/fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c-kube-api-access-l9ql7\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:49:39.314571 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:39.314548 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce96a591-4dcp6" Oct 30 18:49:39.314710 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:39.314536 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce96a591-4dcp6" event={"ID":"fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c","Type":"ContainerDied","Data":"cbb9a3b5c54bd24e1cbe5152c1d020e38bd510e508a8487a868b504868b04cfc"} Oct 30 18:49:39.314710 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:39.314670 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbb9a3b5c54bd24e1cbe5152c1d020e38bd510e508a8487a868b504868b04cfc" Oct 30 18:49:40.859035 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:40.859000 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a174c9f-2w24v"] Oct 30 18:49:40.860314 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:40.860288 4531 cpu_manager.go:414] "containerMap: removing container" podUID="fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c" containerName="revoke-keys" Oct 30 18:49:40.860451 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:40.860439 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c" containerName="revoke-keys" Oct 30 18:49:40.860734 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:40.860720 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c" containerName="revoke-keys" Oct 30 18:49:40.864971 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:40.864948 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a174c9f-2w24v" Oct 30 18:49:40.917280 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:40.917227 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:49:40.966411 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:40.966389 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:49:40.966556 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:40.966500 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:49:40.966556 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:40.966551 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:49:40.968434 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:40.968411 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a174c9f-2w24v"] Oct 30 18:49:40.986602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:40.986576 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbb7c4d1-e974-4b29-bca0-73f3791b378b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a174c9f-2w24v\" (UID: \"cbb7c4d1-e974-4b29-bca0-73f3791b378b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a174c9f-2w24v" Oct 30 18:49:40.986686 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:40.986607 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6z4b\" (UniqueName: \"kubernetes.io/projected/cbb7c4d1-e974-4b29-bca0-73f3791b378b-kube-api-access-j6z4b\") pod \"maas-api-revoke-sub-1a174c9f-2w24v\" (UID: \"cbb7c4d1-e974-4b29-bca0-73f3791b378b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a174c9f-2w24v" Oct 30 18:49:40.986744 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:40.986696 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbb7c4d1-e974-4b29-bca0-73f3791b378b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a174c9f-2w24v\" (UID: \"cbb7c4d1-e974-4b29-bca0-73f3791b378b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a174c9f-2w24v" Oct 30 18:49:41.087719 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:41.087690 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbb7c4d1-e974-4b29-bca0-73f3791b378b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a174c9f-2w24v\" (UID: \"cbb7c4d1-e974-4b29-bca0-73f3791b378b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a174c9f-2w24v" Oct 30 18:49:41.087870 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:41.087789 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbb7c4d1-e974-4b29-bca0-73f3791b378b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a174c9f-2w24v\" (UID: \"cbb7c4d1-e974-4b29-bca0-73f3791b378b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a174c9f-2w24v" Oct 30 18:49:41.087870 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:41.087809 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6z4b\" (UniqueName: \"kubernetes.io/projected/cbb7c4d1-e974-4b29-bca0-73f3791b378b-kube-api-access-j6z4b\") pod \"maas-api-revoke-sub-1a174c9f-2w24v\" (UID: \"cbb7c4d1-e974-4b29-bca0-73f3791b378b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a174c9f-2w24v" Oct 30 18:49:41.088530 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:41.088511 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbb7c4d1-e974-4b29-bca0-73f3791b378b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a174c9f-2w24v\" (UID: \"cbb7c4d1-e974-4b29-bca0-73f3791b378b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a174c9f-2w24v" Oct 30 18:49:41.090229 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:41.090209 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbb7c4d1-e974-4b29-bca0-73f3791b378b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a174c9f-2w24v\" (UID: \"cbb7c4d1-e974-4b29-bca0-73f3791b378b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a174c9f-2w24v" Oct 30 18:49:41.148614 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:41.148550 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6z4b\" (UniqueName: \"kubernetes.io/projected/cbb7c4d1-e974-4b29-bca0-73f3791b378b-kube-api-access-j6z4b\") pod \"maas-api-revoke-sub-1a174c9f-2w24v\" (UID: \"cbb7c4d1-e974-4b29-bca0-73f3791b378b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a174c9f-2w24v" Oct 30 18:49:41.177397 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:41.177373 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a174c9f-2w24v" Oct 30 18:49:41.548278 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:41.548250 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a174c9f-2w24v"] Oct 30 18:49:41.551331 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:49:41.551304 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb7c4d1_e974_4b29_bca0_73f3791b378b.slice/crio-7c338ef778814520bde09c13c079fd18faa338f313d5985d881e92e4bee33b7a WatchSource:0}: Error finding container 7c338ef778814520bde09c13c079fd18faa338f313d5985d881e92e4bee33b7a: Status 404 returned error can't find the container with id 7c338ef778814520bde09c13c079fd18faa338f313d5985d881e92e4bee33b7a Oct 30 18:49:42.329732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:42.329705 4531 generic.go:356] "Generic (PLEG): container finished" podID="cbb7c4d1-e974-4b29-bca0-73f3791b378b" containerID="922ef74e2aead53317444145cab344d77df5fad113c4df01b346e0a5391c4758" exitCode=0 Oct 30 18:49:42.330029 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:42.329763 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a174c9f-2w24v" event={"ID":"cbb7c4d1-e974-4b29-bca0-73f3791b378b","Type":"ContainerDied","Data":"922ef74e2aead53317444145cab344d77df5fad113c4df01b346e0a5391c4758"} Oct 30 18:49:42.330029 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:42.329804 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a174c9f-2w24v" event={"ID":"cbb7c4d1-e974-4b29-bca0-73f3791b378b","Type":"ContainerStarted","Data":"7c338ef778814520bde09c13c079fd18faa338f313d5985d881e92e4bee33b7a"} Oct 30 18:49:43.466947 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:43.466921 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a174c9f-2w24v" Oct 30 18:49:43.611078 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:43.611002 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbb7c4d1-e974-4b29-bca0-73f3791b378b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbb7c4d1-e974-4b29-bca0-73f3791b378b-maas-api-openshift-ca\") pod \"cbb7c4d1-e974-4b29-bca0-73f3791b378b\" (UID: \"cbb7c4d1-e974-4b29-bca0-73f3791b378b\") " Oct 30 18:49:43.611078 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:43.611060 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbb7c4d1-e974-4b29-bca0-73f3791b378b-kube-api-access-j6z4b\" (UniqueName: \"kubernetes.io/projected/cbb7c4d1-e974-4b29-bca0-73f3791b378b-kube-api-access-j6z4b\") pod \"cbb7c4d1-e974-4b29-bca0-73f3791b378b\" (UID: \"cbb7c4d1-e974-4b29-bca0-73f3791b378b\") " Oct 30 18:49:43.611335 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:43.611120 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbb7c4d1-e974-4b29-bca0-73f3791b378b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbb7c4d1-e974-4b29-bca0-73f3791b378b-maas-api-xks-ca\") pod \"cbb7c4d1-e974-4b29-bca0-73f3791b378b\" (UID: \"cbb7c4d1-e974-4b29-bca0-73f3791b378b\") " Oct 30 18:49:43.611437 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:43.611410 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbb7c4d1-e974-4b29-bca0-73f3791b378b-maas-api-openshift-ca" pod "cbb7c4d1-e974-4b29-bca0-73f3791b378b" (UID: "cbb7c4d1-e974-4b29-bca0-73f3791b378b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:49:43.613322 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:43.613299 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbb7c4d1-e974-4b29-bca0-73f3791b378b-maas-api-xks-ca" pod "cbb7c4d1-e974-4b29-bca0-73f3791b378b" (UID: "cbb7c4d1-e974-4b29-bca0-73f3791b378b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:49:43.613426 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:43.613327 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb7c4d1-e974-4b29-bca0-73f3791b378b-kube-api-access-j6z4b" pod "cbb7c4d1-e974-4b29-bca0-73f3791b378b" (UID: "cbb7c4d1-e974-4b29-bca0-73f3791b378b"). InnerVolumeSpecName "kube-api-access-j6z4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:49:43.712440 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:43.712412 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbb7c4d1-e974-4b29-bca0-73f3791b378b-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:49:43.712440 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:43.712435 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6z4b\" (UniqueName: \"kubernetes.io/projected/cbb7c4d1-e974-4b29-bca0-73f3791b378b-kube-api-access-j6z4b\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:49:43.712613 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:43.712447 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbb7c4d1-e974-4b29-bca0-73f3791b378b-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:49:44.340859 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:44.340826 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a174c9f-2w24v" event={"ID":"cbb7c4d1-e974-4b29-bca0-73f3791b378b","Type":"ContainerDied","Data":"7c338ef778814520bde09c13c079fd18faa338f313d5985d881e92e4bee33b7a"} Oct 30 18:49:44.341025 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:44.340872 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c338ef778814520bde09c13c079fd18faa338f313d5985d881e92e4bee33b7a" Oct 30 18:49:44.341025 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:44.340835 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a174c9f-2w24v" Oct 30 18:49:51.662684 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:51.662648 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08dc48c-s8w9l"] Oct 30 18:49:51.663082 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:51.662990 4531 cpu_manager.go:414] "containerMap: removing container" podUID="cbb7c4d1-e974-4b29-bca0-73f3791b378b" containerName="revoke-keys" Oct 30 18:49:51.663082 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:51.663001 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbb7c4d1-e974-4b29-bca0-73f3791b378b" containerName="revoke-keys" Oct 30 18:49:51.663082 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:51.663080 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbb7c4d1-e974-4b29-bca0-73f3791b378b" containerName="revoke-keys" Oct 30 18:49:51.666044 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:51.666028 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dc48c-s8w9l" Oct 30 18:49:51.718096 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:51.718074 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:49:51.767235 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:51.767197 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:49:51.767235 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:51.767209 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:49:51.767235 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:51.767234 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:49:51.769176 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:51.769158 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08dc48c-s8w9l"] Oct 30 18:49:51.781387 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:51.781354 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px7vx\" (UniqueName: \"kubernetes.io/projected/935bde25-694f-43fb-9874-91a4e5710360-kube-api-access-px7vx\") pod \"maas-api-revoke-sub-f08dc48c-s8w9l\" (UID: \"935bde25-694f-43fb-9874-91a4e5710360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dc48c-s8w9l" Oct 30 18:49:51.781503 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:51.781487 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/935bde25-694f-43fb-9874-91a4e5710360-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f08dc48c-s8w9l\" (UID: \"935bde25-694f-43fb-9874-91a4e5710360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dc48c-s8w9l" Oct 30 18:49:51.781557 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:51.781517 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/935bde25-694f-43fb-9874-91a4e5710360-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f08dc48c-s8w9l\" (UID: \"935bde25-694f-43fb-9874-91a4e5710360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dc48c-s8w9l" Oct 30 18:49:51.882190 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:51.882155 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px7vx\" (UniqueName: \"kubernetes.io/projected/935bde25-694f-43fb-9874-91a4e5710360-kube-api-access-px7vx\") pod \"maas-api-revoke-sub-f08dc48c-s8w9l\" (UID: \"935bde25-694f-43fb-9874-91a4e5710360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dc48c-s8w9l" Oct 30 18:49:51.882329 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:51.882241 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/935bde25-694f-43fb-9874-91a4e5710360-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f08dc48c-s8w9l\" (UID: \"935bde25-694f-43fb-9874-91a4e5710360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dc48c-s8w9l" Oct 30 18:49:51.882329 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:51.882263 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/935bde25-694f-43fb-9874-91a4e5710360-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f08dc48c-s8w9l\" (UID: \"935bde25-694f-43fb-9874-91a4e5710360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dc48c-s8w9l" Oct 30 18:49:51.882847 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:51.882817 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/935bde25-694f-43fb-9874-91a4e5710360-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f08dc48c-s8w9l\" (UID: \"935bde25-694f-43fb-9874-91a4e5710360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dc48c-s8w9l" Oct 30 18:49:51.884656 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:51.884636 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/935bde25-694f-43fb-9874-91a4e5710360-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f08dc48c-s8w9l\" (UID: \"935bde25-694f-43fb-9874-91a4e5710360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dc48c-s8w9l" Oct 30 18:49:51.982574 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:51.982502 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px7vx\" (UniqueName: \"kubernetes.io/projected/935bde25-694f-43fb-9874-91a4e5710360-kube-api-access-px7vx\") pod \"maas-api-revoke-sub-f08dc48c-s8w9l\" (UID: \"935bde25-694f-43fb-9874-91a4e5710360\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dc48c-s8w9l" Oct 30 18:49:52.278245 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:52.278215 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dc48c-s8w9l" Oct 30 18:49:52.651974 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:52.651914 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08dc48c-s8w9l"] Oct 30 18:49:52.654318 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:49:52.654278 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod935bde25_694f_43fb_9874_91a4e5710360.slice/crio-ff71e7f8adfa0b0042c9a3cf8b02f71cf3af82c4fa2ce5d2e6013ecbcfeea0e3 WatchSource:0}: Error finding container ff71e7f8adfa0b0042c9a3cf8b02f71cf3af82c4fa2ce5d2e6013ecbcfeea0e3: Status 404 returned error can't find the container with id ff71e7f8adfa0b0042c9a3cf8b02f71cf3af82c4fa2ce5d2e6013ecbcfeea0e3 Oct 30 18:49:53.386321 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:53.386285 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dc48c-s8w9l" event={"ID":"935bde25-694f-43fb-9874-91a4e5710360","Type":"ContainerStarted","Data":"4d88e5607b1c13d224401e9890e27570443ee0a4ec428487bcdbde83c02647ed"} Oct 30 18:49:53.386678 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:53.386342 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dc48c-s8w9l" event={"ID":"935bde25-694f-43fb-9874-91a4e5710360","Type":"ContainerStarted","Data":"ff71e7f8adfa0b0042c9a3cf8b02f71cf3af82c4fa2ce5d2e6013ecbcfeea0e3"} Oct 30 18:49:53.503792 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:53.503591 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dc48c-s8w9l" podStartSLOduration=2.503575536 podStartE2EDuration="2.503575536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:53.502907115 +0000 UTC m=+953.530012075" watchObservedRunningTime="2026-10-30 18:49:53.503575536 +0000 UTC m=+953.530680499" Oct 30 18:49:54.392919 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:54.392841 4531 generic.go:356] "Generic (PLEG): container finished" podID="935bde25-694f-43fb-9874-91a4e5710360" containerID="4d88e5607b1c13d224401e9890e27570443ee0a4ec428487bcdbde83c02647ed" exitCode=0 Oct 30 18:49:54.393283 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:54.392926 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dc48c-s8w9l" event={"ID":"935bde25-694f-43fb-9874-91a4e5710360","Type":"ContainerDied","Data":"4d88e5607b1c13d224401e9890e27570443ee0a4ec428487bcdbde83c02647ed"} Oct 30 18:49:55.534598 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:55.534574 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dc48c-s8w9l" Oct 30 18:49:55.619451 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:55.619422 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/935bde25-694f-43fb-9874-91a4e5710360-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/935bde25-694f-43fb-9874-91a4e5710360-maas-api-xks-ca\") pod \"935bde25-694f-43fb-9874-91a4e5710360\" (UID: \"935bde25-694f-43fb-9874-91a4e5710360\") " Oct 30 18:49:55.619592 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:55.619503 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/935bde25-694f-43fb-9874-91a4e5710360-kube-api-access-px7vx\" (UniqueName: \"kubernetes.io/projected/935bde25-694f-43fb-9874-91a4e5710360-kube-api-access-px7vx\") pod \"935bde25-694f-43fb-9874-91a4e5710360\" (UID: \"935bde25-694f-43fb-9874-91a4e5710360\") " Oct 30 18:49:55.619592 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:55.619525 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/935bde25-694f-43fb-9874-91a4e5710360-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/935bde25-694f-43fb-9874-91a4e5710360-maas-api-openshift-ca\") pod \"935bde25-694f-43fb-9874-91a4e5710360\" (UID: \"935bde25-694f-43fb-9874-91a4e5710360\") " Oct 30 18:49:55.619942 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:55.619911 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935bde25-694f-43fb-9874-91a4e5710360-maas-api-openshift-ca" pod "935bde25-694f-43fb-9874-91a4e5710360" (UID: "935bde25-694f-43fb-9874-91a4e5710360"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:49:55.621629 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:55.621597 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/935bde25-694f-43fb-9874-91a4e5710360-maas-api-xks-ca" pod "935bde25-694f-43fb-9874-91a4e5710360" (UID: "935bde25-694f-43fb-9874-91a4e5710360"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:49:55.621841 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:55.621823 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935bde25-694f-43fb-9874-91a4e5710360-kube-api-access-px7vx" pod "935bde25-694f-43fb-9874-91a4e5710360" (UID: "935bde25-694f-43fb-9874-91a4e5710360"). InnerVolumeSpecName "kube-api-access-px7vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:49:55.720420 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:55.720354 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px7vx\" (UniqueName: \"kubernetes.io/projected/935bde25-694f-43fb-9874-91a4e5710360-kube-api-access-px7vx\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:49:55.720420 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:55.720378 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/935bde25-694f-43fb-9874-91a4e5710360-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:49:55.720420 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:55.720388 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/935bde25-694f-43fb-9874-91a4e5710360-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:49:56.397880 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:49:56.397789 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:49:56.404605 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:56.404578 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dc48c-s8w9l" Oct 30 18:49:56.404726 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:56.404578 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08dc48c-s8w9l" event={"ID":"935bde25-694f-43fb-9874-91a4e5710360","Type":"ContainerDied","Data":"ff71e7f8adfa0b0042c9a3cf8b02f71cf3af82c4fa2ce5d2e6013ecbcfeea0e3"} Oct 30 18:49:56.404726 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:49:56.404718 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff71e7f8adfa0b0042c9a3cf8b02f71cf3af82c4fa2ce5d2e6013ecbcfeea0e3" Oct 30 18:50:09.255929 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.255889 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9"] Oct 30 18:50:09.256498 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.256326 4531 cpu_manager.go:414] "containerMap: removing container" podUID="935bde25-694f-43fb-9874-91a4e5710360" containerName="revoke-keys" Oct 30 18:50:09.256498 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.256340 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="935bde25-694f-43fb-9874-91a4e5710360" containerName="revoke-keys" Oct 30 18:50:09.256498 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.256414 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="935bde25-694f-43fb-9874-91a4e5710360" containerName="revoke-keys" Oct 30 18:50:09.259882 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.259856 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.341497 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.341055 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9"] Oct 30 18:50:09.347192 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.347123 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.347395 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.347201 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrjsk\" (UniqueName: \"kubernetes.io/projected/7d74e373-44cb-4895-b98f-7efa9be01b7d-kube-api-access-xrjsk\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.347395 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.347224 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.347395 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.347251 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.347395 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.347282 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.347625 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.347397 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.347625 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.347451 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d74e373-44cb-4895-b98f-7efa9be01b7d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.347625 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.347549 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.347625 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.347591 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.448833 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.448786 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.448833 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.448836 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.449119 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.448865 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.449119 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.448895 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d74e373-44cb-4895-b98f-7efa9be01b7d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.449119 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.448962 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.449119 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.448994 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.449119 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.449068 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.449119 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.449100 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrjsk\" (UniqueName: \"kubernetes.io/projected/7d74e373-44cb-4895-b98f-7efa9be01b7d-kube-api-access-xrjsk\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.449489 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.449173 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.449489 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.449345 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-workload-socket\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.449489 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.449460 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-workload-certs\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.449653 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.449536 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-credential-socket\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.449873 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.449850 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d74e373-44cb-4895-b98f-7efa9be01b7d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.449954 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.449864 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-data\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.451530 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.451504 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-envoy\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.451848 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.451827 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.510193 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.510081 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-token\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.510375 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.510306 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrjsk\" (UniqueName: \"kubernetes.io/projected/7d74e373-44cb-4895-b98f-7efa9be01b7d-kube-api-access-xrjsk\") pod \"maas-default-gateway-openshift-default-66db998887-wqgt9\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:09.573559 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:09.573516 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:10.022775 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:10.022748 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9"] Oct 30 18:50:10.025337 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:50:10.025303 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d74e373_44cb_4895_b98f_7efa9be01b7d.slice/crio-f5bcbe5aac3111182e359fe224f9775381e7a91bccd33a57dce139d120083fbd WatchSource:0}: Error finding container f5bcbe5aac3111182e359fe224f9775381e7a91bccd33a57dce139d120083fbd: Status 404 returned error can't find the container with id f5bcbe5aac3111182e359fe224f9775381e7a91bccd33a57dce139d120083fbd Oct 30 18:50:10.027848 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:10.027805 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:50:10.027968 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:10.027885 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:50:10.027968 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:10.027917 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:50:10.476768 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:10.476668 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" event={"ID":"7d74e373-44cb-4895-b98f-7efa9be01b7d","Type":"ContainerStarted","Data":"fb4efa242bc08b5c770431eabef642fe8223899960947925df2ebc7180183b6e"} Oct 30 18:50:10.476768 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:10.476747 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" event={"ID":"7d74e373-44cb-4895-b98f-7efa9be01b7d","Type":"ContainerStarted","Data":"f5bcbe5aac3111182e359fe224f9775381e7a91bccd33a57dce139d120083fbd"} Oct 30 18:50:10.573989 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:10.573945 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:10.642455 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:10.642390 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" podStartSLOduration=1.642368703 podStartE2EDuration="1.642368703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:10.598610328 +0000 UTC m=+970.625715296" watchObservedRunningTime="2026-10-30 18:50:10.642368703 +0000 UTC m=+970.669473662" Oct 30 18:50:11.576106 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:11.576070 4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-wqgt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 18:50:11.576533 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:11.576141 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" podUID="7d74e373-44cb-4895-b98f-7efa9be01b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 18:50:12.574076 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:12.574030 4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-wqgt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 18:50:12.574310 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:12.574105 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" podUID="7d74e373-44cb-4895-b98f-7efa9be01b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 18:50:13.574600 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:13.574555 4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-wqgt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 18:50:13.575005 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:13.574617 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" podUID="7d74e373-44cb-4895-b98f-7efa9be01b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 18:50:14.235156 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:14.232809 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba9d1ff8-dhrxj"] Oct 30 18:50:14.243467 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:14.243436 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9d1ff8-dhrxj" Oct 30 18:50:14.291076 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:14.291042 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba9d1ff8-dhrxj"] Oct 30 18:50:14.295911 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:14.295883 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:50:14.295911 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:14.295885 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:50:14.296109 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:14.296054 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:50:14.298601 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:14.298576 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00d659b2-ee5c-43e9-a2e5-82bbbee24f0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba9d1ff8-dhrxj\" (UID: \"00d659b2-ee5c-43e9-a2e5-82bbbee24f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9d1ff8-dhrxj" Oct 30 18:50:14.298732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:14.298711 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dzjd\" (UniqueName: \"kubernetes.io/projected/00d659b2-ee5c-43e9-a2e5-82bbbee24f0c-kube-api-access-9dzjd\") pod \"maas-api-revoke-sub-ba9d1ff8-dhrxj\" (UID: \"00d659b2-ee5c-43e9-a2e5-82bbbee24f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9d1ff8-dhrxj" Oct 30 18:50:14.298789 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:14.298754 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00d659b2-ee5c-43e9-a2e5-82bbbee24f0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba9d1ff8-dhrxj\" (UID: \"00d659b2-ee5c-43e9-a2e5-82bbbee24f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9d1ff8-dhrxj" Oct 30 18:50:14.341461 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:14.341413 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:50:14.400345 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:14.400311 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00d659b2-ee5c-43e9-a2e5-82bbbee24f0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba9d1ff8-dhrxj\" (UID: \"00d659b2-ee5c-43e9-a2e5-82bbbee24f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9d1ff8-dhrxj" Oct 30 18:50:14.400526 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:14.400380 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dzjd\" (UniqueName: \"kubernetes.io/projected/00d659b2-ee5c-43e9-a2e5-82bbbee24f0c-kube-api-access-9dzjd\") pod \"maas-api-revoke-sub-ba9d1ff8-dhrxj\" (UID: \"00d659b2-ee5c-43e9-a2e5-82bbbee24f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9d1ff8-dhrxj" Oct 30 18:50:14.400526 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:14.400415 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00d659b2-ee5c-43e9-a2e5-82bbbee24f0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba9d1ff8-dhrxj\" (UID: \"00d659b2-ee5c-43e9-a2e5-82bbbee24f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9d1ff8-dhrxj" Oct 30 18:50:14.400973 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:14.400952 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00d659b2-ee5c-43e9-a2e5-82bbbee24f0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba9d1ff8-dhrxj\" (UID: \"00d659b2-ee5c-43e9-a2e5-82bbbee24f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9d1ff8-dhrxj" Oct 30 18:50:14.403018 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:14.402995 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00d659b2-ee5c-43e9-a2e5-82bbbee24f0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba9d1ff8-dhrxj\" (UID: \"00d659b2-ee5c-43e9-a2e5-82bbbee24f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9d1ff8-dhrxj" Oct 30 18:50:14.462758 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:14.462731 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dzjd\" (UniqueName: \"kubernetes.io/projected/00d659b2-ee5c-43e9-a2e5-82bbbee24f0c-kube-api-access-9dzjd\") pod \"maas-api-revoke-sub-ba9d1ff8-dhrxj\" (UID: \"00d659b2-ee5c-43e9-a2e5-82bbbee24f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9d1ff8-dhrxj" Oct 30 18:50:14.557379 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:14.557334 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9d1ff8-dhrxj" Oct 30 18:50:14.574519 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:14.574489 4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-wqgt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 18:50:14.574643 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:14.574551 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" podUID="7d74e373-44cb-4895-b98f-7efa9be01b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 18:50:14.945240 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:14.945213 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba9d1ff8-dhrxj"] Oct 30 18:50:14.947953 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:50:14.947923 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d659b2_ee5c_43e9_a2e5_82bbbee24f0c.slice/crio-5170899b5470a43d97f1ec50b5108b8d5c31dba2639b91693d33bb86436361bd WatchSource:0}: Error finding container 5170899b5470a43d97f1ec50b5108b8d5c31dba2639b91693d33bb86436361bd: Status 404 returned error can't find the container with id 5170899b5470a43d97f1ec50b5108b8d5c31dba2639b91693d33bb86436361bd Oct 30 18:50:15.506601 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:15.506562 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9d1ff8-dhrxj" event={"ID":"00d659b2-ee5c-43e9-a2e5-82bbbee24f0c","Type":"ContainerStarted","Data":"59e71809aa2e928b3cfffbc89b62198e3313400d9cf9e9e5cf166455508efc08"} Oct 30 18:50:15.506797 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:15.506628 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9d1ff8-dhrxj" event={"ID":"00d659b2-ee5c-43e9-a2e5-82bbbee24f0c","Type":"ContainerStarted","Data":"5170899b5470a43d97f1ec50b5108b8d5c31dba2639b91693d33bb86436361bd"} Oct 30 18:50:15.574813 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:15.574772 4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-wqgt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 18:50:15.575325 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:15.574836 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" podUID="7d74e373-44cb-4895-b98f-7efa9be01b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 18:50:15.646227 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:15.645261 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9d1ff8-dhrxj" podStartSLOduration=1.6452401650000001 podStartE2EDuration="1.645240165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:15.642667743 +0000 UTC m=+975.669772703" watchObservedRunningTime="2026-10-30 18:50:15.645240165 +0000 UTC m=+975.672345126" Oct 30 18:50:16.517154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:16.516735 4531 generic.go:356] "Generic (PLEG): container finished" podID="00d659b2-ee5c-43e9-a2e5-82bbbee24f0c" containerID="59e71809aa2e928b3cfffbc89b62198e3313400d9cf9e9e5cf166455508efc08" exitCode=0 Oct 30 18:50:16.517154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:16.516807 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9d1ff8-dhrxj" event={"ID":"00d659b2-ee5c-43e9-a2e5-82bbbee24f0c","Type":"ContainerDied","Data":"59e71809aa2e928b3cfffbc89b62198e3313400d9cf9e9e5cf166455508efc08"} Oct 30 18:50:16.574011 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:16.573969 4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-wqgt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 18:50:16.574215 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:16.574043 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" podUID="7d74e373-44cb-4895-b98f-7efa9be01b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 18:50:17.593259 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:17.593212 4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-wqgt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 18:50:17.593782 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:17.593286 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" podUID="7d74e373-44cb-4895-b98f-7efa9be01b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 18:50:18.011303 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:18.011264 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9d1ff8-dhrxj" Oct 30 18:50:18.185780 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:18.185669 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00d659b2-ee5c-43e9-a2e5-82bbbee24f0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00d659b2-ee5c-43e9-a2e5-82bbbee24f0c-maas-api-xks-ca\") pod \"00d659b2-ee5c-43e9-a2e5-82bbbee24f0c\" (UID: \"00d659b2-ee5c-43e9-a2e5-82bbbee24f0c\") " Oct 30 18:50:18.186095 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:18.186072 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00d659b2-ee5c-43e9-a2e5-82bbbee24f0c-kube-api-access-9dzjd\" (UniqueName: \"kubernetes.io/projected/00d659b2-ee5c-43e9-a2e5-82bbbee24f0c-kube-api-access-9dzjd\") pod \"00d659b2-ee5c-43e9-a2e5-82bbbee24f0c\" (UID: \"00d659b2-ee5c-43e9-a2e5-82bbbee24f0c\") " Oct 30 18:50:18.186365 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:18.186345 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00d659b2-ee5c-43e9-a2e5-82bbbee24f0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00d659b2-ee5c-43e9-a2e5-82bbbee24f0c-maas-api-openshift-ca\") pod \"00d659b2-ee5c-43e9-a2e5-82bbbee24f0c\" (UID: \"00d659b2-ee5c-43e9-a2e5-82bbbee24f0c\") " Oct 30 18:50:18.187218 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:18.187192 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00d659b2-ee5c-43e9-a2e5-82bbbee24f0c-maas-api-openshift-ca" pod "00d659b2-ee5c-43e9-a2e5-82bbbee24f0c" (UID: "00d659b2-ee5c-43e9-a2e5-82bbbee24f0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:50:18.190470 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:18.190416 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00d659b2-ee5c-43e9-a2e5-82bbbee24f0c-maas-api-xks-ca" pod "00d659b2-ee5c-43e9-a2e5-82bbbee24f0c" (UID: "00d659b2-ee5c-43e9-a2e5-82bbbee24f0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:50:18.190732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:18.190695 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d659b2-ee5c-43e9-a2e5-82bbbee24f0c-kube-api-access-9dzjd" pod "00d659b2-ee5c-43e9-a2e5-82bbbee24f0c" (UID: "00d659b2-ee5c-43e9-a2e5-82bbbee24f0c"). InnerVolumeSpecName "kube-api-access-9dzjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:50:18.287729 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:18.287664 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dzjd\" (UniqueName: \"kubernetes.io/projected/00d659b2-ee5c-43e9-a2e5-82bbbee24f0c-kube-api-access-9dzjd\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:50:18.287729 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:18.287691 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00d659b2-ee5c-43e9-a2e5-82bbbee24f0c-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:50:18.287729 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:18.287703 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00d659b2-ee5c-43e9-a2e5-82bbbee24f0c-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:50:18.537169 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:18.536374 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9d1ff8-dhrxj" event={"ID":"00d659b2-ee5c-43e9-a2e5-82bbbee24f0c","Type":"ContainerDied","Data":"5170899b5470a43d97f1ec50b5108b8d5c31dba2639b91693d33bb86436361bd"} Oct 30 18:50:18.537169 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:18.536442 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5170899b5470a43d97f1ec50b5108b8d5c31dba2639b91693d33bb86436361bd" Oct 30 18:50:18.537169 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:18.536532 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9d1ff8-dhrxj" Oct 30 18:50:19.569771 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:19.569734 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:19.570374 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:19.570345 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:19.571270 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:50:19.571246 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 18:50:26.407723 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:50:26.407676 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:50:56.457162 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:50:56.453375 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:51:01.627307 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:01.627272 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx"] Oct 30 18:51:01.627786 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:01.627652 4531 cpu_manager.go:414] "containerMap: removing container" podUID="00d659b2-ee5c-43e9-a2e5-82bbbee24f0c" containerName="revoke-keys" Oct 30 18:51:01.627786 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:01.627663 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00d659b2-ee5c-43e9-a2e5-82bbbee24f0c" containerName="revoke-keys" Oct 30 18:51:01.627786 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:01.627744 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="00d659b2-ee5c-43e9-a2e5-82bbbee24f0c" containerName="revoke-keys" Oct 30 18:51:01.630910 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:01.630892 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 18:51:01.732430 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:01.732388 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fd61e3\"/\"distinct-w1-fd61e3-kserve-self-signed-certs\"" Oct 30 18:51:01.734765 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:01.734742 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx"] Oct 30 18:51:01.764525 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:01.764492 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2rvw\" (UniqueName: \"kubernetes.io/projected/f23b88e6-d3b4-4975-818a-22cee9d30dce-kube-api-access-t2rvw\") pod \"distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx\" (UID: \"f23b88e6-d3b4-4975-818a-22cee9d30dce\") " pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 18:51:01.764698 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:01.764551 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-model-cache\") pod \"distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx\" (UID: \"f23b88e6-d3b4-4975-818a-22cee9d30dce\") " pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 18:51:01.764698 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:01.764641 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f23b88e6-d3b4-4975-818a-22cee9d30dce-tls-certs\") pod \"distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx\" (UID: \"f23b88e6-d3b4-4975-818a-22cee9d30dce\") " pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 18:51:01.764772 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:01.764730 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-tmp-dir\") pod \"distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx\" (UID: \"f23b88e6-d3b4-4975-818a-22cee9d30dce\") " pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 18:51:01.764852 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:01.764834 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-home\") pod \"distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx\" (UID: \"f23b88e6-d3b4-4975-818a-22cee9d30dce\") " pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 18:51:01.764893 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:01.764860 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-dshm\") pod \"distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx\" (UID: \"f23b88e6-d3b4-4975-818a-22cee9d30dce\") " pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 18:51:01.866446 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:01.866406 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-home\") pod \"distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx\" (UID: \"f23b88e6-d3b4-4975-818a-22cee9d30dce\") " pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 18:51:01.866446 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:01.866449 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-dshm\") pod \"distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx\" (UID: \"f23b88e6-d3b4-4975-818a-22cee9d30dce\") " pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 18:51:01.866725 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:01.866499 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2rvw\" (UniqueName: \"kubernetes.io/projected/f23b88e6-d3b4-4975-818a-22cee9d30dce-kube-api-access-t2rvw\") pod \"distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx\" (UID: \"f23b88e6-d3b4-4975-818a-22cee9d30dce\") " pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 18:51:01.866725 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:01.866546 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-model-cache\") pod \"distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx\" (UID: \"f23b88e6-d3b4-4975-818a-22cee9d30dce\") " pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 18:51:01.866725 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:01.866569 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f23b88e6-d3b4-4975-818a-22cee9d30dce-tls-certs\") pod \"distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx\" (UID: \"f23b88e6-d3b4-4975-818a-22cee9d30dce\") " pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 18:51:01.866725 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:01.866706 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-tmp-dir\") pod \"distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx\" (UID: \"f23b88e6-d3b4-4975-818a-22cee9d30dce\") " pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 18:51:01.866995 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:01.866917 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-home\") pod \"distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx\" (UID: \"f23b88e6-d3b4-4975-818a-22cee9d30dce\") " pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 18:51:01.866995 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:01.866947 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-model-cache\") pod \"distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx\" (UID: \"f23b88e6-d3b4-4975-818a-22cee9d30dce\") " pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 18:51:01.867148 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:01.867110 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-tmp-dir\") pod \"distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx\" (UID: \"f23b88e6-d3b4-4975-818a-22cee9d30dce\") " pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 18:51:01.869144 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:01.869102 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-dshm\") pod \"distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx\" (UID: \"f23b88e6-d3b4-4975-818a-22cee9d30dce\") " pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 18:51:01.869405 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:01.869388 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f23b88e6-d3b4-4975-818a-22cee9d30dce-tls-certs\") pod \"distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx\" (UID: \"f23b88e6-d3b4-4975-818a-22cee9d30dce\") " pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 18:51:01.928698 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:01.928605 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2rvw\" (UniqueName: \"kubernetes.io/projected/f23b88e6-d3b4-4975-818a-22cee9d30dce-kube-api-access-t2rvw\") pod \"distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx\" (UID: \"f23b88e6-d3b4-4975-818a-22cee9d30dce\") " pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 18:51:01.943214 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:01.943175 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 18:51:02.343872 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:02.343847 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx"] Oct 30 18:51:02.346427 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:51:02.346397 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf23b88e6_d3b4_4975_818a_22cee9d30dce.slice/crio-c5df9b46440a384e8d165fb78ff22282b5b153cc01872cff0ced57490c554634 WatchSource:0}: Error finding container c5df9b46440a384e8d165fb78ff22282b5b153cc01872cff0ced57490c554634: Status 404 returned error can't find the container with id c5df9b46440a384e8d165fb78ff22282b5b153cc01872cff0ced57490c554634 Oct 30 18:51:02.809410 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:02.809374 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" event={"ID":"f23b88e6-d3b4-4975-818a-22cee9d30dce","Type":"ContainerStarted","Data":"8350fe722af29da10c0821946defc1c6e67f386563499b68f05fa2386ec5aee8"} Oct 30 18:51:02.809410 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:02.809417 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" event={"ID":"f23b88e6-d3b4-4975-818a-22cee9d30dce","Type":"ContainerStarted","Data":"c5df9b46440a384e8d165fb78ff22282b5b153cc01872cff0ced57490c554634"} Oct 30 18:51:02.929001 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:02.928931 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" podStartSLOduration=1.9289076870000001 podStartE2EDuration="1.928907687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:02.925656183 +0000 UTC m=+1022.952761156" watchObservedRunningTime="2026-10-30 18:51:02.928907687 +0000 UTC m=+1022.956012647" Oct 30 18:51:02.944179 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:02.944114 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 18:51:11.943556 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:11.943462 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 18:51:11.956521 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:11.956488 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 18:51:12.884525 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:12.884492 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 18:51:26.399549 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:51:26.399506 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:51:35.598851 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:35.598820 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-tfmtb-7bqnn"] Oct 30 18:51:35.602743 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:35.602717 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tfmtb-7bqnn" Oct 30 18:51:35.655209 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:35.655178 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 18:51:35.704433 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:35.704401 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lwjtg\"" Oct 30 18:51:35.741819 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:35.741773 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-trusted-ca-bundle\") pod \"periodic-gathering-tfmtb-7bqnn\" (UID: \"e20d0816-9f25-4e7f-bf65-56a30e7a05eb\") " pod="openshift-insights/periodic-gathering-tfmtb-7bqnn" Oct 30 18:51:35.742034 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:35.741832 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-archives-path\") pod \"periodic-gathering-tfmtb-7bqnn\" (UID: \"e20d0816-9f25-4e7f-bf65-56a30e7a05eb\") " pod="openshift-insights/periodic-gathering-tfmtb-7bqnn" Oct 30 18:51:35.742034 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:35.741951 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-service-ca-bundle\") pod \"periodic-gathering-tfmtb-7bqnn\" (UID: \"e20d0816-9f25-4e7f-bf65-56a30e7a05eb\") " pod="openshift-insights/periodic-gathering-tfmtb-7bqnn" Oct 30 18:51:35.742034 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:35.742028 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6f4t\" (UniqueName: \"kubernetes.io/projected/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-kube-api-access-x6f4t\") pod \"periodic-gathering-tfmtb-7bqnn\" (UID: \"e20d0816-9f25-4e7f-bf65-56a30e7a05eb\") " pod="openshift-insights/periodic-gathering-tfmtb-7bqnn" Oct 30 18:51:35.804623 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:35.804586 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 18:51:35.806680 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:35.806655 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-tfmtb-7bqnn"] Oct 30 18:51:35.842733 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:35.842696 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6f4t\" (UniqueName: \"kubernetes.io/projected/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-kube-api-access-x6f4t\") pod \"periodic-gathering-tfmtb-7bqnn\" (UID: \"e20d0816-9f25-4e7f-bf65-56a30e7a05eb\") " pod="openshift-insights/periodic-gathering-tfmtb-7bqnn" Oct 30 18:51:35.842916 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:35.842757 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-trusted-ca-bundle\") pod \"periodic-gathering-tfmtb-7bqnn\" (UID: \"e20d0816-9f25-4e7f-bf65-56a30e7a05eb\") " pod="openshift-insights/periodic-gathering-tfmtb-7bqnn" Oct 30 18:51:35.842916 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:35.842804 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-archives-path\") pod \"periodic-gathering-tfmtb-7bqnn\" (UID: \"e20d0816-9f25-4e7f-bf65-56a30e7a05eb\") " pod="openshift-insights/periodic-gathering-tfmtb-7bqnn" Oct 30 18:51:35.842916 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:35.842840 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-service-ca-bundle\") pod \"periodic-gathering-tfmtb-7bqnn\" (UID: \"e20d0816-9f25-4e7f-bf65-56a30e7a05eb\") " pod="openshift-insights/periodic-gathering-tfmtb-7bqnn" Oct 30 18:51:35.844249 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:35.843655 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-archives-path\") pod \"periodic-gathering-tfmtb-7bqnn\" (UID: \"e20d0816-9f25-4e7f-bf65-56a30e7a05eb\") " pod="openshift-insights/periodic-gathering-tfmtb-7bqnn" Oct 30 18:51:35.844249 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:35.844113 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-service-ca-bundle\") pod \"periodic-gathering-tfmtb-7bqnn\" (UID: \"e20d0816-9f25-4e7f-bf65-56a30e7a05eb\") " pod="openshift-insights/periodic-gathering-tfmtb-7bqnn" Oct 30 18:51:35.844249 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:35.844191 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-trusted-ca-bundle\") pod \"periodic-gathering-tfmtb-7bqnn\" (UID: \"e20d0816-9f25-4e7f-bf65-56a30e7a05eb\") " pod="openshift-insights/periodic-gathering-tfmtb-7bqnn" Oct 30 18:51:35.904875 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:35.904795 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6f4t\" (UniqueName: \"kubernetes.io/projected/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-kube-api-access-x6f4t\") pod \"periodic-gathering-tfmtb-7bqnn\" (UID: \"e20d0816-9f25-4e7f-bf65-56a30e7a05eb\") " pod="openshift-insights/periodic-gathering-tfmtb-7bqnn" Oct 30 18:51:35.915631 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:35.915600 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tfmtb-7bqnn" Oct 30 18:51:36.298212 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:36.298183 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-tfmtb-7bqnn"] Oct 30 18:51:36.300568 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:51:36.300535 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode20d0816_9f25_4e7f_bf65_56a30e7a05eb.slice/crio-26568c18b600e86b12cdfb1c3907447817195e1889c0d7b0af879fd35a53e84b WatchSource:0}: Error finding container 26568c18b600e86b12cdfb1c3907447817195e1889c0d7b0af879fd35a53e84b: Status 404 returned error can't find the container with id 26568c18b600e86b12cdfb1c3907447817195e1889c0d7b0af879fd35a53e84b Oct 30 18:51:37.052021 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:37.051980 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tfmtb-7bqnn" event={"ID":"e20d0816-9f25-4e7f-bf65-56a30e7a05eb","Type":"ContainerStarted","Data":"224c24b30098c77267c37b5512244d9afdcebc2bb36ee94e796258adf5671053"} Oct 30 18:51:37.052548 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:37.052036 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tfmtb-7bqnn" event={"ID":"e20d0816-9f25-4e7f-bf65-56a30e7a05eb","Type":"ContainerStarted","Data":"26568c18b600e86b12cdfb1c3907447817195e1889c0d7b0af879fd35a53e84b"} Oct 30 18:51:37.167987 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:37.167921 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-tfmtb-7bqnn" podStartSLOduration=2.167904122 podStartE2EDuration="2.167904122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:37.167661117 +0000 UTC m=+1057.194766076" watchObservedRunningTime="2026-10-30 18:51:37.167904122 +0000 UTC m=+1057.195009081" Oct 30 18:51:44.516288 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:44.516254 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x"] Oct 30 18:51:44.520490 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:44.520472 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" Oct 30 18:51:44.527687 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:44.527657 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-model-cache\") pod \"e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x\" (UID: \"5686e156-9f19-401b-b305-99acd65a6ce5\") " pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" Oct 30 18:51:44.527803 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:44.527725 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-dshm\") pod \"e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x\" (UID: \"5686e156-9f19-401b-b305-99acd65a6ce5\") " pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" Oct 30 18:51:44.527803 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:44.527753 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5686e156-9f19-401b-b305-99acd65a6ce5-tls-certs\") pod \"e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x\" (UID: \"5686e156-9f19-401b-b305-99acd65a6ce5\") " pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" Oct 30 18:51:44.527803 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:44.527791 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt45h\" (UniqueName: \"kubernetes.io/projected/5686e156-9f19-401b-b305-99acd65a6ce5-kube-api-access-lt45h\") pod \"e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x\" (UID: \"5686e156-9f19-401b-b305-99acd65a6ce5\") " pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" Oct 30 18:51:44.527966 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:44.527833 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-tmp-dir\") pod \"e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x\" (UID: \"5686e156-9f19-401b-b305-99acd65a6ce5\") " pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" Oct 30 18:51:44.527966 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:44.527860 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-home\") pod \"e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x\" (UID: \"5686e156-9f19-401b-b305-99acd65a6ce5\") " pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" Oct 30 18:51:44.578266 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:44.578227 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x"] Oct 30 18:51:44.624354 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:44.624327 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2ade09\"/\"e2e-conflict-b-d331d324-kserve-self-signed-certs\"" Oct 30 18:51:44.629422 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:44.629397 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-model-cache\") pod \"e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x\" (UID: \"5686e156-9f19-401b-b305-99acd65a6ce5\") " pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" Oct 30 18:51:44.629552 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:44.629435 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-dshm\") pod \"e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x\" (UID: \"5686e156-9f19-401b-b305-99acd65a6ce5\") " pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" Oct 30 18:51:44.629552 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:44.629453 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5686e156-9f19-401b-b305-99acd65a6ce5-tls-certs\") pod \"e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x\" (UID: \"5686e156-9f19-401b-b305-99acd65a6ce5\") " pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" Oct 30 18:51:44.629552 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:44.629482 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt45h\" (UniqueName: \"kubernetes.io/projected/5686e156-9f19-401b-b305-99acd65a6ce5-kube-api-access-lt45h\") pod \"e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x\" (UID: \"5686e156-9f19-401b-b305-99acd65a6ce5\") " pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" Oct 30 18:51:44.629552 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:44.629517 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-tmp-dir\") pod \"e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x\" (UID: \"5686e156-9f19-401b-b305-99acd65a6ce5\") " pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" Oct 30 18:51:44.629552 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:44.629542 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-home\") pod \"e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x\" (UID: \"5686e156-9f19-401b-b305-99acd65a6ce5\") " pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" Oct 30 18:51:44.629943 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:44.629918 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-model-cache\") pod \"e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x\" (UID: \"5686e156-9f19-401b-b305-99acd65a6ce5\") " pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" Oct 30 18:51:44.630085 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:44.630050 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-tmp-dir\") pod \"e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x\" (UID: \"5686e156-9f19-401b-b305-99acd65a6ce5\") " pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" Oct 30 18:51:44.630405 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:44.630210 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-home\") pod \"e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x\" (UID: \"5686e156-9f19-401b-b305-99acd65a6ce5\") " pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" Oct 30 18:51:44.632040 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:44.632015 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-dshm\") pod \"e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x\" (UID: \"5686e156-9f19-401b-b305-99acd65a6ce5\") " pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" Oct 30 18:51:44.632369 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:44.632349 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5686e156-9f19-401b-b305-99acd65a6ce5-tls-certs\") pod \"e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x\" (UID: \"5686e156-9f19-401b-b305-99acd65a6ce5\") " pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" Oct 30 18:51:44.691480 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:44.691453 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt45h\" (UniqueName: \"kubernetes.io/projected/5686e156-9f19-401b-b305-99acd65a6ce5-kube-api-access-lt45h\") pod \"e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x\" (UID: \"5686e156-9f19-401b-b305-99acd65a6ce5\") " pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" Oct 30 18:51:44.833108 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:44.833013 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" Oct 30 18:51:45.233270 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:45.233246 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x"] Oct 30 18:51:45.235078 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:51:45.235046 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5686e156_9f19_401b_b305_99acd65a6ce5.slice/crio-e07d225956abfb17462dde0f72f5f838f98c95f2c5bd5896cfb3ffc10ddfaf2b WatchSource:0}: Error finding container e07d225956abfb17462dde0f72f5f838f98c95f2c5bd5896cfb3ffc10ddfaf2b: Status 404 returned error can't find the container with id e07d225956abfb17462dde0f72f5f838f98c95f2c5bd5896cfb3ffc10ddfaf2b Oct 30 18:51:46.101422 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:46.101379 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" event={"ID":"5686e156-9f19-401b-b305-99acd65a6ce5","Type":"ContainerStarted","Data":"e872f062eb13641426f14160fe714988385615574e6bd8fb5e02ea87c1d1cb32"} Oct 30 18:51:46.101422 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:46.101432 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" event={"ID":"5686e156-9f19-401b-b305-99acd65a6ce5","Type":"ContainerStarted","Data":"e07d225956abfb17462dde0f72f5f838f98c95f2c5bd5896cfb3ffc10ddfaf2b"} Oct 30 18:51:46.219236 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:46.219178 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" podStartSLOduration=2.219160892 podStartE2EDuration="2.219160892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:46.21783925 +0000 UTC m=+1066.244944208" watchObservedRunningTime="2026-10-30 18:51:46.219160892 +0000 UTC m=+1066.246265851" Oct 30 18:51:46.833638 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:46.833605 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" Oct 30 18:51:54.833497 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:54.833449 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" Oct 30 18:51:54.846625 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:54.846595 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" Oct 30 18:51:55.159106 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:51:55.159031 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" Oct 30 18:51:56.401162 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:51:56.401100 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:52:00.218820 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:00.218783 4531 scope.go:122] "RemoveContainer" containerID="adcc63ff33ca6bcc86bc0c949842e0a95c02ed885bb4de346bd92b7da525a368" Oct 30 18:52:00.228427 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:00.228407 4531 scope.go:122] "RemoveContainer" containerID="36a467da2c7acf74ade90624cbdfcaa857c6e3fe12397ac329dfed87383b5433" Oct 30 18:52:00.237095 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:00.237073 4531 scope.go:122] "RemoveContainer" containerID="fd101a80fc008ee6357cc696e58c83dfe27e3bb4609fa134095e46e9104658b7" Oct 30 18:52:00.245517 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:00.245496 4531 scope.go:122] "RemoveContainer" containerID="cd71bea467af19c3f2cfc2f7ce3b84dd210e7815c611d93f92ed98a49541c45c" Oct 30 18:52:01.055412 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:01.055378 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3477ba14-xjrlt"] Oct 30 18:52:01.060738 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:01.060718 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3477ba14-xjrlt" Oct 30 18:52:01.115081 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:01.115052 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:52:01.164350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:01.164316 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:52:01.164350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:01.164336 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:52:01.164558 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:01.164429 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:52:01.166540 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:01.166522 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3477ba14-xjrlt"] Oct 30 18:52:01.196044 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:01.196009 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7913969-bc03-4c2a-8d9a-986941af9924-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3477ba14-xjrlt\" (UID: \"c7913969-bc03-4c2a-8d9a-986941af9924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3477ba14-xjrlt" Oct 30 18:52:01.196256 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:01.196173 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7913969-bc03-4c2a-8d9a-986941af9924-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3477ba14-xjrlt\" (UID: \"c7913969-bc03-4c2a-8d9a-986941af9924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3477ba14-xjrlt" Oct 30 18:52:01.196256 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:01.196204 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkztp\" (UniqueName: \"kubernetes.io/projected/c7913969-bc03-4c2a-8d9a-986941af9924-kube-api-access-pkztp\") pod \"maas-api-revoke-sub-3477ba14-xjrlt\" (UID: \"c7913969-bc03-4c2a-8d9a-986941af9924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3477ba14-xjrlt" Oct 30 18:52:01.297058 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:01.297012 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7913969-bc03-4c2a-8d9a-986941af9924-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3477ba14-xjrlt\" (UID: \"c7913969-bc03-4c2a-8d9a-986941af9924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3477ba14-xjrlt" Oct 30 18:52:01.297518 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:01.297068 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkztp\" (UniqueName: \"kubernetes.io/projected/c7913969-bc03-4c2a-8d9a-986941af9924-kube-api-access-pkztp\") pod \"maas-api-revoke-sub-3477ba14-xjrlt\" (UID: \"c7913969-bc03-4c2a-8d9a-986941af9924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3477ba14-xjrlt" Oct 30 18:52:01.297518 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:01.297115 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7913969-bc03-4c2a-8d9a-986941af9924-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3477ba14-xjrlt\" (UID: \"c7913969-bc03-4c2a-8d9a-986941af9924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3477ba14-xjrlt" Oct 30 18:52:01.297930 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:01.297897 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7913969-bc03-4c2a-8d9a-986941af9924-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3477ba14-xjrlt\" (UID: \"c7913969-bc03-4c2a-8d9a-986941af9924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3477ba14-xjrlt" Oct 30 18:52:01.300575 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:01.300548 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7913969-bc03-4c2a-8d9a-986941af9924-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3477ba14-xjrlt\" (UID: \"c7913969-bc03-4c2a-8d9a-986941af9924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3477ba14-xjrlt" Oct 30 18:52:01.359267 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:01.359183 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkztp\" (UniqueName: \"kubernetes.io/projected/c7913969-bc03-4c2a-8d9a-986941af9924-kube-api-access-pkztp\") pod \"maas-api-revoke-sub-3477ba14-xjrlt\" (UID: \"c7913969-bc03-4c2a-8d9a-986941af9924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3477ba14-xjrlt" Oct 30 18:52:01.373453 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:01.373428 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3477ba14-xjrlt" Oct 30 18:52:01.759382 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:01.759353 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3477ba14-xjrlt"] Oct 30 18:52:01.761607 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:52:01.761582 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7913969_bc03_4c2a_8d9a_986941af9924.slice/crio-4ef4039339d96e7b569f1538cd54a4fe64b88c9cad08530323f2566fe962901e WatchSource:0}: Error finding container 4ef4039339d96e7b569f1538cd54a4fe64b88c9cad08530323f2566fe962901e: Status 404 returned error can't find the container with id 4ef4039339d96e7b569f1538cd54a4fe64b88c9cad08530323f2566fe962901e Oct 30 18:52:02.194785 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:02.194750 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3477ba14-xjrlt" event={"ID":"c7913969-bc03-4c2a-8d9a-986941af9924","Type":"ContainerStarted","Data":"4adcf418f32352900890efee8b9a9823a94aaa8b14292f9d2fb2609696c03bbb"} Oct 30 18:52:02.194980 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:02.194798 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3477ba14-xjrlt" event={"ID":"c7913969-bc03-4c2a-8d9a-986941af9924","Type":"ContainerStarted","Data":"4ef4039339d96e7b569f1538cd54a4fe64b88c9cad08530323f2566fe962901e"} Oct 30 18:52:02.310242 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:02.310092 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3477ba14-xjrlt" podStartSLOduration=1.310078873 podStartE2EDuration="1.310078873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:02.309293355 +0000 UTC m=+1082.336398313" watchObservedRunningTime="2026-10-30 18:52:02.310078873 +0000 UTC m=+1082.337183832" Oct 30 18:52:03.201755 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:03.201721 4531 generic.go:356] "Generic (PLEG): container finished" podID="c7913969-bc03-4c2a-8d9a-986941af9924" containerID="4adcf418f32352900890efee8b9a9823a94aaa8b14292f9d2fb2609696c03bbb" exitCode=0 Oct 30 18:52:03.201934 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:03.201778 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3477ba14-xjrlt" event={"ID":"c7913969-bc03-4c2a-8d9a-986941af9924","Type":"ContainerDied","Data":"4adcf418f32352900890efee8b9a9823a94aaa8b14292f9d2fb2609696c03bbb"} Oct 30 18:52:04.357770 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:04.357742 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3477ba14-xjrlt" Oct 30 18:52:04.426010 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:04.425969 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7913969-bc03-4c2a-8d9a-986941af9924-kube-api-access-pkztp\" (UniqueName: \"kubernetes.io/projected/c7913969-bc03-4c2a-8d9a-986941af9924-kube-api-access-pkztp\") pod \"c7913969-bc03-4c2a-8d9a-986941af9924\" (UID: \"c7913969-bc03-4c2a-8d9a-986941af9924\") " Oct 30 18:52:04.426196 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:04.426023 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7913969-bc03-4c2a-8d9a-986941af9924-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7913969-bc03-4c2a-8d9a-986941af9924-maas-api-openshift-ca\") pod \"c7913969-bc03-4c2a-8d9a-986941af9924\" (UID: \"c7913969-bc03-4c2a-8d9a-986941af9924\") " Oct 30 18:52:04.426196 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:04.426058 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7913969-bc03-4c2a-8d9a-986941af9924-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7913969-bc03-4c2a-8d9a-986941af9924-maas-api-xks-ca\") pod \"c7913969-bc03-4c2a-8d9a-986941af9924\" (UID: \"c7913969-bc03-4c2a-8d9a-986941af9924\") " Oct 30 18:52:04.426509 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:04.426479 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7913969-bc03-4c2a-8d9a-986941af9924-maas-api-openshift-ca" pod "c7913969-bc03-4c2a-8d9a-986941af9924" (UID: "c7913969-bc03-4c2a-8d9a-986941af9924"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:04.428738 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:04.428709 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7913969-bc03-4c2a-8d9a-986941af9924-kube-api-access-pkztp" pod "c7913969-bc03-4c2a-8d9a-986941af9924" (UID: "c7913969-bc03-4c2a-8d9a-986941af9924"). InnerVolumeSpecName "kube-api-access-pkztp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:04.428865 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:04.428748 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7913969-bc03-4c2a-8d9a-986941af9924-maas-api-xks-ca" pod "c7913969-bc03-4c2a-8d9a-986941af9924" (UID: "c7913969-bc03-4c2a-8d9a-986941af9924"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:04.527451 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:04.527421 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkztp\" (UniqueName: \"kubernetes.io/projected/c7913969-bc03-4c2a-8d9a-986941af9924-kube-api-access-pkztp\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:52:04.527451 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:04.527448 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7913969-bc03-4c2a-8d9a-986941af9924-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:52:04.527451 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:04.527458 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7913969-bc03-4c2a-8d9a-986941af9924-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:52:05.216888 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:05.216861 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3477ba14-xjrlt" Oct 30 18:52:05.217068 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:05.216858 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3477ba14-xjrlt" event={"ID":"c7913969-bc03-4c2a-8d9a-986941af9924","Type":"ContainerDied","Data":"4ef4039339d96e7b569f1538cd54a4fe64b88c9cad08530323f2566fe962901e"} Oct 30 18:52:05.217068 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:05.216996 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ef4039339d96e7b569f1538cd54a4fe64b88c9cad08530323f2566fe962901e" Oct 30 18:52:15.335953 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:15.335914 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e41ba5fe-nhxtk"] Oct 30 18:52:15.337211 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:15.337188 4531 cpu_manager.go:414] "containerMap: removing container" podUID="c7913969-bc03-4c2a-8d9a-986941af9924" containerName="revoke-keys" Oct 30 18:52:15.337350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:15.337338 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7913969-bc03-4c2a-8d9a-986941af9924" containerName="revoke-keys" Oct 30 18:52:15.337570 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:15.337557 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7913969-bc03-4c2a-8d9a-986941af9924" containerName="revoke-keys" Oct 30 18:52:15.344450 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:15.344424 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41ba5fe-nhxtk" Oct 30 18:52:15.396762 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:15.396726 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e41ba5fe-nhxtk"] Oct 30 18:52:15.401684 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:15.401653 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:52:15.402090 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:15.402059 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:52:15.402240 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:15.402102 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:52:15.402303 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:15.402238 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:52:15.458059 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:15.458011 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70fb0b0c-7934-41b8-ae2a-5155338da606-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e41ba5fe-nhxtk\" (UID: \"70fb0b0c-7934-41b8-ae2a-5155338da606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41ba5fe-nhxtk" Oct 30 18:52:15.458059 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:15.458062 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mskb6\" (UniqueName: \"kubernetes.io/projected/70fb0b0c-7934-41b8-ae2a-5155338da606-kube-api-access-mskb6\") pod \"maas-api-revoke-sub-e41ba5fe-nhxtk\" (UID: \"70fb0b0c-7934-41b8-ae2a-5155338da606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41ba5fe-nhxtk" Oct 30 18:52:15.458311 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:15.458227 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70fb0b0c-7934-41b8-ae2a-5155338da606-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e41ba5fe-nhxtk\" (UID: \"70fb0b0c-7934-41b8-ae2a-5155338da606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41ba5fe-nhxtk" Oct 30 18:52:15.559441 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:15.559402 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70fb0b0c-7934-41b8-ae2a-5155338da606-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e41ba5fe-nhxtk\" (UID: \"70fb0b0c-7934-41b8-ae2a-5155338da606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41ba5fe-nhxtk" Oct 30 18:52:15.559628 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:15.559500 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70fb0b0c-7934-41b8-ae2a-5155338da606-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e41ba5fe-nhxtk\" (UID: \"70fb0b0c-7934-41b8-ae2a-5155338da606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41ba5fe-nhxtk" Oct 30 18:52:15.559628 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:15.559538 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mskb6\" (UniqueName: \"kubernetes.io/projected/70fb0b0c-7934-41b8-ae2a-5155338da606-kube-api-access-mskb6\") pod \"maas-api-revoke-sub-e41ba5fe-nhxtk\" (UID: \"70fb0b0c-7934-41b8-ae2a-5155338da606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41ba5fe-nhxtk" Oct 30 18:52:15.560348 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:15.560317 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70fb0b0c-7934-41b8-ae2a-5155338da606-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e41ba5fe-nhxtk\" (UID: \"70fb0b0c-7934-41b8-ae2a-5155338da606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41ba5fe-nhxtk" Oct 30 18:52:15.562160 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:15.562107 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70fb0b0c-7934-41b8-ae2a-5155338da606-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e41ba5fe-nhxtk\" (UID: \"70fb0b0c-7934-41b8-ae2a-5155338da606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41ba5fe-nhxtk" Oct 30 18:52:15.621725 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:15.621646 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mskb6\" (UniqueName: \"kubernetes.io/projected/70fb0b0c-7934-41b8-ae2a-5155338da606-kube-api-access-mskb6\") pod \"maas-api-revoke-sub-e41ba5fe-nhxtk\" (UID: \"70fb0b0c-7934-41b8-ae2a-5155338da606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41ba5fe-nhxtk" Oct 30 18:52:15.663375 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:15.663339 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41ba5fe-nhxtk" Oct 30 18:52:16.039812 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:16.039786 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e41ba5fe-nhxtk"] Oct 30 18:52:16.041811 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:52:16.041782 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70fb0b0c_7934_41b8_ae2a_5155338da606.slice/crio-9b25d40d860558f322180ea0d4e899c7cdf5a4ffb0518a2df572a71d7acd7071 WatchSource:0}: Error finding container 9b25d40d860558f322180ea0d4e899c7cdf5a4ffb0518a2df572a71d7acd7071: Status 404 returned error can't find the container with id 9b25d40d860558f322180ea0d4e899c7cdf5a4ffb0518a2df572a71d7acd7071 Oct 30 18:52:16.276848 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:16.276807 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41ba5fe-nhxtk" event={"ID":"70fb0b0c-7934-41b8-ae2a-5155338da606","Type":"ContainerStarted","Data":"9b25d40d860558f322180ea0d4e899c7cdf5a4ffb0518a2df572a71d7acd7071"} Oct 30 18:52:17.284181 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:17.284148 4531 generic.go:356] "Generic (PLEG): container finished" podID="70fb0b0c-7934-41b8-ae2a-5155338da606" containerID="ca4ed686aa1d4920ccaee8a2c68d015e59f98b86946f39b070e0baa4dc60935f" exitCode=0 Oct 30 18:52:17.284589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:17.284215 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41ba5fe-nhxtk" event={"ID":"70fb0b0c-7934-41b8-ae2a-5155338da606","Type":"ContainerDied","Data":"ca4ed686aa1d4920ccaee8a2c68d015e59f98b86946f39b070e0baa4dc60935f"} Oct 30 18:52:18.432904 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:18.432875 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41ba5fe-nhxtk" Oct 30 18:52:18.592669 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:18.592567 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70fb0b0c-7934-41b8-ae2a-5155338da606-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70fb0b0c-7934-41b8-ae2a-5155338da606-maas-api-openshift-ca\") pod \"70fb0b0c-7934-41b8-ae2a-5155338da606\" (UID: \"70fb0b0c-7934-41b8-ae2a-5155338da606\") " Oct 30 18:52:18.592669 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:18.592658 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70fb0b0c-7934-41b8-ae2a-5155338da606-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70fb0b0c-7934-41b8-ae2a-5155338da606-maas-api-xks-ca\") pod \"70fb0b0c-7934-41b8-ae2a-5155338da606\" (UID: \"70fb0b0c-7934-41b8-ae2a-5155338da606\") " Oct 30 18:52:18.592908 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:18.592805 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70fb0b0c-7934-41b8-ae2a-5155338da606-kube-api-access-mskb6\" (UniqueName: \"kubernetes.io/projected/70fb0b0c-7934-41b8-ae2a-5155338da606-kube-api-access-mskb6\") pod \"70fb0b0c-7934-41b8-ae2a-5155338da606\" (UID: \"70fb0b0c-7934-41b8-ae2a-5155338da606\") " Oct 30 18:52:18.592961 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:18.592919 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70fb0b0c-7934-41b8-ae2a-5155338da606-maas-api-openshift-ca" pod "70fb0b0c-7934-41b8-ae2a-5155338da606" (UID: "70fb0b0c-7934-41b8-ae2a-5155338da606"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:18.593171 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:18.593118 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70fb0b0c-7934-41b8-ae2a-5155338da606-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:52:18.594940 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:18.594903 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70fb0b0c-7934-41b8-ae2a-5155338da606-maas-api-xks-ca" pod "70fb0b0c-7934-41b8-ae2a-5155338da606" (UID: "70fb0b0c-7934-41b8-ae2a-5155338da606"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:18.595059 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:18.594973 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70fb0b0c-7934-41b8-ae2a-5155338da606-kube-api-access-mskb6" pod "70fb0b0c-7934-41b8-ae2a-5155338da606" (UID: "70fb0b0c-7934-41b8-ae2a-5155338da606"). InnerVolumeSpecName "kube-api-access-mskb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:18.694747 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:18.694700 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70fb0b0c-7934-41b8-ae2a-5155338da606-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:52:18.694747 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:18.694735 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mskb6\" (UniqueName: \"kubernetes.io/projected/70fb0b0c-7934-41b8-ae2a-5155338da606-kube-api-access-mskb6\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:52:19.295102 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:19.295067 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41ba5fe-nhxtk" event={"ID":"70fb0b0c-7934-41b8-ae2a-5155338da606","Type":"ContainerDied","Data":"9b25d40d860558f322180ea0d4e899c7cdf5a4ffb0518a2df572a71d7acd7071"} Oct 30 18:52:19.295102 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:19.295099 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41ba5fe-nhxtk" Oct 30 18:52:19.295375 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:19.295118 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b25d40d860558f322180ea0d4e899c7cdf5a4ffb0518a2df572a71d7acd7071" Oct 30 18:52:23.688238 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.688201 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj"] Oct 30 18:52:23.688644 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.688589 4531 cpu_manager.go:414] "containerMap: removing container" podUID="70fb0b0c-7934-41b8-ae2a-5155338da606" containerName="revoke-keys" Oct 30 18:52:23.688644 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.688601 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70fb0b0c-7934-41b8-ae2a-5155338da606" containerName="revoke-keys" Oct 30 18:52:23.688718 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.688694 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="70fb0b0c-7934-41b8-ae2a-5155338da606" containerName="revoke-keys" Oct 30 18:52:23.692269 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.692234 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.743295 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.743246 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-envoy\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.743295 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.743295 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtwp6\" (UniqueName: \"kubernetes.io/projected/2b5e06df-6296-4e17-8dfb-2c02e919e622-kube-api-access-xtwp6\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.743572 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.743313 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-credential-socket\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.743572 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.743466 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-podinfo\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.743572 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.743530 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-workload-certs\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.743706 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.743610 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-data\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.743706 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.743653 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-workload-socket\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.743812 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.743749 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5e06df-6296-4e17-8dfb-2c02e919e622-istiod-ca-cert\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.743812 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.743793 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-token\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.744482 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.744466 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2ade09-openshift-default-dockercfg-8v89k\"" Oct 30 18:52:23.797362 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.797317 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj"] Oct 30 18:52:23.844777 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.844733 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-data\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.844975 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.844796 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-workload-socket\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.844975 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.844850 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5e06df-6296-4e17-8dfb-2c02e919e622-istiod-ca-cert\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.844975 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.844887 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-token\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.844975 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.844938 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-envoy\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.844975 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.844965 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtwp6\" (UniqueName: \"kubernetes.io/projected/2b5e06df-6296-4e17-8dfb-2c02e919e622-kube-api-access-xtwp6\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.845509 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.844991 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-credential-socket\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.845509 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.845033 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-podinfo\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.845509 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.845086 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-workload-certs\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.845509 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.845195 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-data\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.845509 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.845426 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-workload-socket\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.845509 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.845487 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-workload-certs\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.845815 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.845587 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-credential-socket\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.846262 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.846238 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5e06df-6296-4e17-8dfb-2c02e919e622-istiod-ca-cert\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.847501 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.847480 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-envoy\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.848079 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.848059 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-podinfo\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.905915 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.905884 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-token\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:23.906067 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:23.906010 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtwp6\" (UniqueName: \"kubernetes.io/projected/2b5e06df-6296-4e17-8dfb-2c02e919e622-kube-api-access-xtwp6\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:24.005436 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:24.005405 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:24.433975 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:24.433950 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj"] Oct 30 18:52:24.435826 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:52:24.435795 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5e06df_6296_4e17_8dfb_2c02e919e622.slice/crio-04e49c2a812dca714a8c47a4321e2cf40a65bb7783fd3b0a16d91fb0a5f3c2c8 WatchSource:0}: Error finding container 04e49c2a812dca714a8c47a4321e2cf40a65bb7783fd3b0a16d91fb0a5f3c2c8: Status 404 returned error can't find the container with id 04e49c2a812dca714a8c47a4321e2cf40a65bb7783fd3b0a16d91fb0a5f3c2c8 Oct 30 18:52:24.438023 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:24.437993 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:52:24.438099 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:24.438058 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:52:24.438099 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:24.438087 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:52:25.344797 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:25.344717 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" event={"ID":"2b5e06df-6296-4e17-8dfb-2c02e919e622","Type":"ContainerStarted","Data":"5d4a6f32308b9ca1319c373239bb4eb020806da1b1dd32ac70638ec873b428b2"} Oct 30 18:52:25.345564 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:25.344858 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" event={"ID":"2b5e06df-6296-4e17-8dfb-2c02e919e622","Type":"ContainerStarted","Data":"04e49c2a812dca714a8c47a4321e2cf40a65bb7783fd3b0a16d91fb0a5f3c2c8"} Oct 30 18:52:25.466795 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:25.466739 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" podStartSLOduration=2.466724708 podStartE2EDuration="2.466724708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:25.465922034 +0000 UTC m=+1105.493026993" watchObservedRunningTime="2026-10-30 18:52:25.466724708 +0000 UTC m=+1105.493829667" Oct 30 18:52:26.005948 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:26.005914 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:26.007204 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:26.007176 4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:52:26.007340 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:26.007227 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" podUID="2b5e06df-6296-4e17-8dfb-2c02e919e622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 18:52:26.397377 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:52:26.397290 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:52:27.006424 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:27.006388 4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:52:27.006634 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:27.006443 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" podUID="2b5e06df-6296-4e17-8dfb-2c02e919e622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 18:52:27.821154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:27.817615 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv"] Oct 30 18:52:27.859155 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:27.842843 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:27.918155 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:27.893103 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv"] Oct 30 18:52:27.950728 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:27.896606 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9e8996-dad3-4888-821f-2ec713398074-istiod-ca-cert\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:27.950995 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:27.950977 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-istio-data\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:27.951119 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:27.951104 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-workload-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:27.951278 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:27.951257 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-workload-certs\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:27.955278 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:27.955248 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef9e8996-dad3-4888-821f-2ec713398074-istio-token\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:27.963152 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:27.955479 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p6qv\" (UniqueName: \"kubernetes.io/projected/ef9e8996-dad3-4888-821f-2ec713398074-kube-api-access-7p6qv\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:27.972317 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:27.972265 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-istio-envoy\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:27.972528 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:27.972513 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9e8996-dad3-4888-821f-2ec713398074-istio-podinfo\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:27.972685 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:27.972669 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-credential-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:27.972896 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:27.950374 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-fd61e3-openshift-default-dockercfg-tn6nh\"" Oct 30 18:52:28.037287 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:28.037251 4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:52:28.037585 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:28.037545 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" podUID="2b5e06df-6296-4e17-8dfb-2c02e919e622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 18:52:28.095208 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:28.074714 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-credential-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:28.095208 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:28.074782 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9e8996-dad3-4888-821f-2ec713398074-istiod-ca-cert\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:28.095208 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:28.074822 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-istio-data\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:28.095208 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:28.074851 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-workload-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:28.095208 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:28.074881 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-workload-certs\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:28.095208 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:28.074943 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef9e8996-dad3-4888-821f-2ec713398074-istio-token\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:28.095208 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:28.074985 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p6qv\" (UniqueName: \"kubernetes.io/projected/ef9e8996-dad3-4888-821f-2ec713398074-kube-api-access-7p6qv\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:28.095208 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:28.075049 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-istio-envoy\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:28.095208 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:28.075083 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9e8996-dad3-4888-821f-2ec713398074-istio-podinfo\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:28.095208 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:28.094737 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-workload-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:28.095208 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:28.095002 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-credential-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:28.095956 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:28.095629 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9e8996-dad3-4888-821f-2ec713398074-istiod-ca-cert\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:28.095956 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:28.095902 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-istio-data\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:28.114342 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:28.096589 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-workload-certs\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:28.114342 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:28.109550 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9e8996-dad3-4888-821f-2ec713398074-istio-podinfo\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:28.164489 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:28.144445 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-istio-envoy\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:28.167234 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:28.167180 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef9e8996-dad3-4888-821f-2ec713398074-istio-token\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:28.251362 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:28.251295 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p6qv\" (UniqueName: \"kubernetes.io/projected/ef9e8996-dad3-4888-821f-2ec713398074-kube-api-access-7p6qv\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:28.541670 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:28.541350 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:29.013160 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:29.010919 4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:52:29.013160 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:29.010979 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" podUID="2b5e06df-6296-4e17-8dfb-2c02e919e622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 18:52:29.576482 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:29.576387 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv"] Oct 30 18:52:29.658816 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:29.657960 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:52:29.658816 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:29.658061 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:52:29.658816 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:29.658104 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:52:30.011161 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:30.007524 4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:52:30.011161 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:30.007587 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" podUID="2b5e06df-6296-4e17-8dfb-2c02e919e622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 18:52:30.397736 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:30.397703 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" event={"ID":"ef9e8996-dad3-4888-821f-2ec713398074","Type":"ContainerStarted","Data":"1de00f85dd2ecd987277d0c1d0b3139fd3116cdb99671f82353d644582c57f1d"} Oct 30 18:52:30.398236 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:30.398215 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" event={"ID":"ef9e8996-dad3-4888-821f-2ec713398074","Type":"ContainerStarted","Data":"7f5ef83295b6eff5e72706064e993575402626651bdde96100790ce408123012"} Oct 30 18:52:30.543337 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:30.543283 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:30.545273 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:30.545233 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:52:30.545472 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:30.545301 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" podUID="ef9e8996-dad3-4888-821f-2ec713398074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 18:52:30.568885 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:30.568830 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" podStartSLOduration=3.568810642 podStartE2EDuration="3.568810642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:30.56691431 +0000 UTC m=+1110.594019270" watchObservedRunningTime="2026-10-30 18:52:30.568810642 +0000 UTC m=+1110.595915654" Oct 30 18:52:31.006708 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:31.006664 4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:52:31.006898 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:31.006719 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" podUID="2b5e06df-6296-4e17-8dfb-2c02e919e622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 18:52:31.542667 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:31.542623 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:52:31.543280 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:31.543235 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" podUID="ef9e8996-dad3-4888-821f-2ec713398074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 18:52:32.006367 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:32.006330 4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:52:32.006549 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:32.006394 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" podUID="2b5e06df-6296-4e17-8dfb-2c02e919e622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 18:52:32.543106 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:32.542629 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:52:32.543106 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:32.542694 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" podUID="ef9e8996-dad3-4888-821f-2ec713398074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 18:52:33.010744 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:33.010714 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:33.011164 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:33.011115 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:33.011799 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:33.011781 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 18:52:33.542545 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:33.542502 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:52:33.542774 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:33.542563 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" podUID="ef9e8996-dad3-4888-821f-2ec713398074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 18:52:34.542706 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:34.542673 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:52:34.543112 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:34.542741 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" podUID="ef9e8996-dad3-4888-821f-2ec713398074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 18:52:35.542740 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:35.542702 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:52:35.543195 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:35.542756 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" podUID="ef9e8996-dad3-4888-821f-2ec713398074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 18:52:36.542464 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:36.542419 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:52:36.542623 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:36.542508 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" podUID="ef9e8996-dad3-4888-821f-2ec713398074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 18:52:37.611757 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:37.611720 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:37.612208 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:37.612057 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:37.612929 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:37.612911 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 18:52:44.270192 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.270096 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s"] Oct 30 18:52:44.273970 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.273949 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.326833 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.326797 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-e7929d-openshift-default-dockercfg-v4s8t\"" Oct 30 18:52:44.353727 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.353694 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-workload-certs\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.353886 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.353737 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1788f5-f0c0-4457-a310-694f47a97d2c-istiod-ca-cert\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.353886 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.353761 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mnqc\" (UniqueName: \"kubernetes.io/projected/ce1788f5-f0c0-4457-a310-694f47a97d2c-kube-api-access-4mnqc\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.353886 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.353863 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-credential-socket\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.353994 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.353907 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-workload-socket\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.353994 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.353956 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-token\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.353994 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.353988 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-data\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.354119 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.354026 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-podinfo\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.354119 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.354073 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-envoy\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.379349 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.379324 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s"] Oct 30 18:52:44.455475 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.455438 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mnqc\" (UniqueName: \"kubernetes.io/projected/ce1788f5-f0c0-4457-a310-694f47a97d2c-kube-api-access-4mnqc\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.455685 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.455498 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-credential-socket\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.455685 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.455525 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-workload-socket\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.455685 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.455579 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-token\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.455685 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.455620 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-data\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.455685 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.455648 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-podinfo\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.455685 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.455682 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-envoy\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.455995 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.455765 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-workload-certs\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.455995 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.455806 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1788f5-f0c0-4457-a310-694f47a97d2c-istiod-ca-cert\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.455995 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.455875 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-credential-socket\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.455995 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.455930 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-workload-socket\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.456280 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.456029 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-data\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.456280 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.456154 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-workload-certs\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.456545 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.456521 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1788f5-f0c0-4457-a310-694f47a97d2c-istiod-ca-cert\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.458212 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.458193 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-envoy\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.458356 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.458337 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-podinfo\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.516783 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.516755 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-token\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.516938 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.516830 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mnqc\" (UniqueName: \"kubernetes.io/projected/ce1788f5-f0c0-4457-a310-694f47a97d2c-kube-api-access-4mnqc\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:44.588051 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:44.587961 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:45.016822 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:45.016797 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s"] Oct 30 18:52:45.018061 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:52:45.018034 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1788f5_f0c0_4457_a310_694f47a97d2c.slice/crio-ae5764b4fe113f7c870869de7eb08a3728fdf305b6d28655d301cb818ba2a9d2 WatchSource:0}: Error finding container ae5764b4fe113f7c870869de7eb08a3728fdf305b6d28655d301cb818ba2a9d2: Status 404 returned error can't find the container with id ae5764b4fe113f7c870869de7eb08a3728fdf305b6d28655d301cb818ba2a9d2 Oct 30 18:52:45.023968 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:45.023934 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:52:45.024053 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:45.024001 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:52:45.024053 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:45.024032 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:52:45.509772 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:45.509735 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" event={"ID":"ce1788f5-f0c0-4457-a310-694f47a97d2c","Type":"ContainerStarted","Data":"16f8c91eb59437589ba4a871644f124d24837dbc9bd29e2735365d9b05e9f6e0"} Oct 30 18:52:45.510202 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:45.509787 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" event={"ID":"ce1788f5-f0c0-4457-a310-694f47a97d2c","Type":"ContainerStarted","Data":"ae5764b4fe113f7c870869de7eb08a3728fdf305b6d28655d301cb818ba2a9d2"} Oct 30 18:52:45.589040 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:45.589013 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:45.635042 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:45.634968 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" podStartSLOduration=1.634945935 podStartE2EDuration="1.634945935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:45.63166651 +0000 UTC m=+1125.658771472" watchObservedRunningTime="2026-10-30 18:52:45.634945935 +0000 UTC m=+1125.662050894" Oct 30 18:52:46.590114 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:46.590079 4531 patch_prober.go:29] interesting pod/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:52:46.590598 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:46.590157 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" podUID="ce1788f5-f0c0-4457-a310-694f47a97d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 18:52:47.589090 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:47.589043 4531 patch_prober.go:29] interesting pod/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:52:47.589329 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:47.589108 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" podUID="ce1788f5-f0c0-4457-a310-694f47a97d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 18:52:48.589264 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:48.589231 4531 patch_prober.go:29] interesting pod/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:52:48.589769 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:48.589297 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" podUID="ce1788f5-f0c0-4457-a310-694f47a97d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 18:52:49.588373 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:49.588340 4531 patch_prober.go:29] interesting pod/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:52:49.588562 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:49.588415 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" podUID="ce1788f5-f0c0-4457-a310-694f47a97d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 18:52:50.588326 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:50.588292 4531 patch_prober.go:29] interesting pod/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:52:50.588816 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:50.588353 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" podUID="ce1788f5-f0c0-4457-a310-694f47a97d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 18:52:51.589198 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:51.589161 4531 patch_prober.go:29] interesting pod/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:52:51.589719 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:51.589223 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" podUID="ce1788f5-f0c0-4457-a310-694f47a97d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 18:52:52.592895 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:52.592868 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:52.593379 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:52.593313 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:52.594042 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:52.594023 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 18:52:53.820919 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:53.820882 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e444fdb-qpbb4"] Oct 30 18:52:53.824918 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:53.824894 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e444fdb-qpbb4" Oct 30 18:52:53.877305 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:53.877272 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:52:53.877305 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:53.877270 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:52:53.926654 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:53.926623 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:52:53.926846 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:53.926795 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:52:53.929349 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:53.929323 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e444fdb-qpbb4"] Oct 30 18:52:53.980686 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:53.980646 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlvmp\" (UniqueName: \"kubernetes.io/projected/04e19546-630e-4c22-8b60-a21bb614c607-kube-api-access-dlvmp\") pod \"maas-api-revoke-sub-6e444fdb-qpbb4\" (UID: \"04e19546-630e-4c22-8b60-a21bb614c607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e444fdb-qpbb4" Oct 30 18:52:53.980869 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:53.980704 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e19546-630e-4c22-8b60-a21bb614c607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e444fdb-qpbb4\" (UID: \"04e19546-630e-4c22-8b60-a21bb614c607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e444fdb-qpbb4" Oct 30 18:52:53.980869 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:53.980782 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e19546-630e-4c22-8b60-a21bb614c607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e444fdb-qpbb4\" (UID: \"04e19546-630e-4c22-8b60-a21bb614c607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e444fdb-qpbb4" Oct 30 18:52:54.082379 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:54.082282 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e19546-630e-4c22-8b60-a21bb614c607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e444fdb-qpbb4\" (UID: \"04e19546-630e-4c22-8b60-a21bb614c607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e444fdb-qpbb4" Oct 30 18:52:54.082537 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:54.082406 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlvmp\" (UniqueName: \"kubernetes.io/projected/04e19546-630e-4c22-8b60-a21bb614c607-kube-api-access-dlvmp\") pod \"maas-api-revoke-sub-6e444fdb-qpbb4\" (UID: \"04e19546-630e-4c22-8b60-a21bb614c607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e444fdb-qpbb4" Oct 30 18:52:54.082537 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:54.082448 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e19546-630e-4c22-8b60-a21bb614c607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e444fdb-qpbb4\" (UID: \"04e19546-630e-4c22-8b60-a21bb614c607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e444fdb-qpbb4" Oct 30 18:52:54.083074 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:54.083050 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e19546-630e-4c22-8b60-a21bb614c607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e444fdb-qpbb4\" (UID: \"04e19546-630e-4c22-8b60-a21bb614c607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e444fdb-qpbb4" Oct 30 18:52:54.084828 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:54.084805 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e19546-630e-4c22-8b60-a21bb614c607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e444fdb-qpbb4\" (UID: \"04e19546-630e-4c22-8b60-a21bb614c607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e444fdb-qpbb4" Oct 30 18:52:54.143956 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:54.143919 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlvmp\" (UniqueName: \"kubernetes.io/projected/04e19546-630e-4c22-8b60-a21bb614c607-kube-api-access-dlvmp\") pod \"maas-api-revoke-sub-6e444fdb-qpbb4\" (UID: \"04e19546-630e-4c22-8b60-a21bb614c607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e444fdb-qpbb4" Oct 30 18:52:54.438506 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:54.438417 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e444fdb-qpbb4" Oct 30 18:52:54.819117 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:54.819082 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e444fdb-qpbb4"] Oct 30 18:52:54.821077 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:52:54.821047 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e19546_630e_4c22_8b60_a21bb614c607.slice/crio-d9bf095ac8123b6f1825487ee369fa0ff6e2f4e156c8bc2c077e62f369903893 WatchSource:0}: Error finding container d9bf095ac8123b6f1825487ee369fa0ff6e2f4e156c8bc2c077e62f369903893: Status 404 returned error can't find the container with id d9bf095ac8123b6f1825487ee369fa0ff6e2f4e156c8bc2c077e62f369903893 Oct 30 18:52:55.591371 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:55.591334 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e444fdb-qpbb4" event={"ID":"04e19546-630e-4c22-8b60-a21bb614c607","Type":"ContainerStarted","Data":"7ade7150efeb1b3737db2bf11b06ab5037004b48bdd306ecac66bb879501c5d4"} Oct 30 18:52:55.591581 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:55.591422 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e444fdb-qpbb4" event={"ID":"04e19546-630e-4c22-8b60-a21bb614c607","Type":"ContainerStarted","Data":"d9bf095ac8123b6f1825487ee369fa0ff6e2f4e156c8bc2c077e62f369903893"} Oct 30 18:52:55.714570 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:55.714506 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e444fdb-qpbb4" podStartSLOduration=2.71448941 podStartE2EDuration="2.71448941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:55.713076824 +0000 UTC m=+1135.740181786" watchObservedRunningTime="2026-10-30 18:52:55.71448941 +0000 UTC m=+1135.741594368" Oct 30 18:52:56.399042 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:52:56.399002 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:52:56.598060 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:56.598028 4531 generic.go:356] "Generic (PLEG): container finished" podID="04e19546-630e-4c22-8b60-a21bb614c607" containerID="7ade7150efeb1b3737db2bf11b06ab5037004b48bdd306ecac66bb879501c5d4" exitCode=0 Oct 30 18:52:56.598257 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:56.598120 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e444fdb-qpbb4" event={"ID":"04e19546-630e-4c22-8b60-a21bb614c607","Type":"ContainerDied","Data":"7ade7150efeb1b3737db2bf11b06ab5037004b48bdd306ecac66bb879501c5d4"} Oct 30 18:52:57.118343 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:57.118311 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x"] Oct 30 18:52:57.759282 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:57.759253 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e444fdb-qpbb4" Oct 30 18:52:57.925667 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:57.925570 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04e19546-630e-4c22-8b60-a21bb614c607-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e19546-630e-4c22-8b60-a21bb614c607-maas-api-openshift-ca\") pod \"04e19546-630e-4c22-8b60-a21bb614c607\" (UID: \"04e19546-630e-4c22-8b60-a21bb614c607\") " Oct 30 18:52:57.925849 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:57.925781 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04e19546-630e-4c22-8b60-a21bb614c607-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e19546-630e-4c22-8b60-a21bb614c607-maas-api-xks-ca\") pod \"04e19546-630e-4c22-8b60-a21bb614c607\" (UID: \"04e19546-630e-4c22-8b60-a21bb614c607\") " Oct 30 18:52:57.925849 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:57.925811 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e19546-630e-4c22-8b60-a21bb614c607-kube-api-access-dlvmp\" (UniqueName: \"kubernetes.io/projected/04e19546-630e-4c22-8b60-a21bb614c607-kube-api-access-dlvmp\") pod \"04e19546-630e-4c22-8b60-a21bb614c607\" (UID: \"04e19546-630e-4c22-8b60-a21bb614c607\") " Oct 30 18:52:57.926008 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:57.925979 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e19546-630e-4c22-8b60-a21bb614c607-maas-api-openshift-ca" pod "04e19546-630e-4c22-8b60-a21bb614c607" (UID: "04e19546-630e-4c22-8b60-a21bb614c607"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:57.926217 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:57.926199 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e19546-630e-4c22-8b60-a21bb614c607-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:52:57.928093 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:57.928067 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04e19546-630e-4c22-8b60-a21bb614c607-maas-api-xks-ca" pod "04e19546-630e-4c22-8b60-a21bb614c607" (UID: "04e19546-630e-4c22-8b60-a21bb614c607"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:57.928221 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:57.928200 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e19546-630e-4c22-8b60-a21bb614c607-kube-api-access-dlvmp" pod "04e19546-630e-4c22-8b60-a21bb614c607" (UID: "04e19546-630e-4c22-8b60-a21bb614c607"). InnerVolumeSpecName "kube-api-access-dlvmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:58.027788 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:58.027750 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e19546-630e-4c22-8b60-a21bb614c607-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:52:58.027788 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:58.027783 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlvmp\" (UniqueName: \"kubernetes.io/projected/04e19546-630e-4c22-8b60-a21bb614c607-kube-api-access-dlvmp\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:52:58.615249 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:58.615219 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e444fdb-qpbb4" Oct 30 18:52:58.615249 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:58.615242 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e444fdb-qpbb4" event={"ID":"04e19546-630e-4c22-8b60-a21bb614c607","Type":"ContainerDied","Data":"d9bf095ac8123b6f1825487ee369fa0ff6e2f4e156c8bc2c077e62f369903893"} Oct 30 18:52:58.615464 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:58.615289 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9bf095ac8123b6f1825487ee369fa0ff6e2f4e156c8bc2c077e62f369903893" Oct 30 18:52:59.750006 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:59.749972 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09ff2cd1-nfggg"] Oct 30 18:52:59.750469 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:59.750455 4531 cpu_manager.go:414] "containerMap: removing container" podUID="04e19546-630e-4c22-8b60-a21bb614c607" containerName="revoke-keys" Oct 30 18:52:59.750513 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:59.750470 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04e19546-630e-4c22-8b60-a21bb614c607" containerName="revoke-keys" Oct 30 18:52:59.750566 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:59.750557 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="04e19546-630e-4c22-8b60-a21bb614c607" containerName="revoke-keys" Oct 30 18:52:59.754027 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:59.754009 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ff2cd1-nfggg" Oct 30 18:52:59.806381 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:59.806345 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:52:59.846711 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:59.846678 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/392fb7ab-dcd4-4b8b-846f-9d94f5b72f73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09ff2cd1-nfggg\" (UID: \"392fb7ab-dcd4-4b8b-846f-9d94f5b72f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ff2cd1-nfggg" Oct 30 18:52:59.846873 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:59.846733 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/392fb7ab-dcd4-4b8b-846f-9d94f5b72f73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09ff2cd1-nfggg\" (UID: \"392fb7ab-dcd4-4b8b-846f-9d94f5b72f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ff2cd1-nfggg" Oct 30 18:52:59.846873 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:59.846828 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h55c\" (UniqueName: \"kubernetes.io/projected/392fb7ab-dcd4-4b8b-846f-9d94f5b72f73-kube-api-access-9h55c\") pod \"maas-api-revoke-sub-09ff2cd1-nfggg\" (UID: \"392fb7ab-dcd4-4b8b-846f-9d94f5b72f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ff2cd1-nfggg" Oct 30 18:52:59.855746 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:59.855717 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:52:59.855888 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:59.855814 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:52:59.855888 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:59.855828 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:52:59.858448 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:59.858421 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09ff2cd1-nfggg"] Oct 30 18:52:59.948297 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:59.948266 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/392fb7ab-dcd4-4b8b-846f-9d94f5b72f73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09ff2cd1-nfggg\" (UID: \"392fb7ab-dcd4-4b8b-846f-9d94f5b72f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ff2cd1-nfggg" Oct 30 18:52:59.948488 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:59.948314 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/392fb7ab-dcd4-4b8b-846f-9d94f5b72f73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09ff2cd1-nfggg\" (UID: \"392fb7ab-dcd4-4b8b-846f-9d94f5b72f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ff2cd1-nfggg" Oct 30 18:52:59.948488 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:59.948334 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h55c\" (UniqueName: \"kubernetes.io/projected/392fb7ab-dcd4-4b8b-846f-9d94f5b72f73-kube-api-access-9h55c\") pod \"maas-api-revoke-sub-09ff2cd1-nfggg\" (UID: \"392fb7ab-dcd4-4b8b-846f-9d94f5b72f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ff2cd1-nfggg" Oct 30 18:52:59.948960 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:59.948934 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/392fb7ab-dcd4-4b8b-846f-9d94f5b72f73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09ff2cd1-nfggg\" (UID: \"392fb7ab-dcd4-4b8b-846f-9d94f5b72f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ff2cd1-nfggg" Oct 30 18:52:59.950872 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:52:59.950849 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/392fb7ab-dcd4-4b8b-846f-9d94f5b72f73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09ff2cd1-nfggg\" (UID: \"392fb7ab-dcd4-4b8b-846f-9d94f5b72f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ff2cd1-nfggg" Oct 30 18:53:00.009735 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:00.009652 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h55c\" (UniqueName: \"kubernetes.io/projected/392fb7ab-dcd4-4b8b-846f-9d94f5b72f73-kube-api-access-9h55c\") pod \"maas-api-revoke-sub-09ff2cd1-nfggg\" (UID: \"392fb7ab-dcd4-4b8b-846f-9d94f5b72f73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ff2cd1-nfggg" Oct 30 18:53:00.118464 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:00.118439 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:53:00.127615 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:00.127583 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ff2cd1-nfggg" Oct 30 18:53:00.508419 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:00.508392 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09ff2cd1-nfggg"] Oct 30 18:53:00.509879 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:53:00.509853 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod392fb7ab_dcd4_4b8b_846f_9d94f5b72f73.slice/crio-8fd180f268c9ef1182f8c49739f18d2f94dd935023ce284f1fcabc2d36d8be4f WatchSource:0}: Error finding container 8fd180f268c9ef1182f8c49739f18d2f94dd935023ce284f1fcabc2d36d8be4f: Status 404 returned error can't find the container with id 8fd180f268c9ef1182f8c49739f18d2f94dd935023ce284f1fcabc2d36d8be4f Oct 30 18:53:00.630272 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:00.630234 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ff2cd1-nfggg" event={"ID":"392fb7ab-dcd4-4b8b-846f-9d94f5b72f73","Type":"ContainerStarted","Data":"8fd180f268c9ef1182f8c49739f18d2f94dd935023ce284f1fcabc2d36d8be4f"} Oct 30 18:53:01.637465 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:01.637435 4531 generic.go:356] "Generic (PLEG): container finished" podID="392fb7ab-dcd4-4b8b-846f-9d94f5b72f73" containerID="3d0ba2afa526285231168a10246f4fffb3a1461b481acfe14a662001466adf47" exitCode=0 Oct 30 18:53:01.637849 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:01.637532 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ff2cd1-nfggg" event={"ID":"392fb7ab-dcd4-4b8b-846f-9d94f5b72f73","Type":"ContainerDied","Data":"3d0ba2afa526285231168a10246f4fffb3a1461b481acfe14a662001466adf47"} Oct 30 18:53:02.783376 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:02.783351 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ff2cd1-nfggg" Oct 30 18:53:02.881304 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:02.881275 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/392fb7ab-dcd4-4b8b-846f-9d94f5b72f73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/392fb7ab-dcd4-4b8b-846f-9d94f5b72f73-maas-api-xks-ca\") pod \"392fb7ab-dcd4-4b8b-846f-9d94f5b72f73\" (UID: \"392fb7ab-dcd4-4b8b-846f-9d94f5b72f73\") " Oct 30 18:53:02.883254 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:02.883233 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/392fb7ab-dcd4-4b8b-846f-9d94f5b72f73-maas-api-xks-ca" pod "392fb7ab-dcd4-4b8b-846f-9d94f5b72f73" (UID: "392fb7ab-dcd4-4b8b-846f-9d94f5b72f73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:02.982165 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:02.982058 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/392fb7ab-dcd4-4b8b-846f-9d94f5b72f73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/392fb7ab-dcd4-4b8b-846f-9d94f5b72f73-maas-api-openshift-ca\") pod \"392fb7ab-dcd4-4b8b-846f-9d94f5b72f73\" (UID: \"392fb7ab-dcd4-4b8b-846f-9d94f5b72f73\") " Oct 30 18:53:02.982165 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:02.982103 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/392fb7ab-dcd4-4b8b-846f-9d94f5b72f73-kube-api-access-9h55c\" (UniqueName: \"kubernetes.io/projected/392fb7ab-dcd4-4b8b-846f-9d94f5b72f73-kube-api-access-9h55c\") pod \"392fb7ab-dcd4-4b8b-846f-9d94f5b72f73\" (UID: \"392fb7ab-dcd4-4b8b-846f-9d94f5b72f73\") " Oct 30 18:53:02.982512 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:02.982487 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/392fb7ab-dcd4-4b8b-846f-9d94f5b72f73-maas-api-openshift-ca" pod "392fb7ab-dcd4-4b8b-846f-9d94f5b72f73" (UID: "392fb7ab-dcd4-4b8b-846f-9d94f5b72f73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:02.982557 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:02.982544 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/392fb7ab-dcd4-4b8b-846f-9d94f5b72f73-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:02.984463 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:02.984446 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392fb7ab-dcd4-4b8b-846f-9d94f5b72f73-kube-api-access-9h55c" pod "392fb7ab-dcd4-4b8b-846f-9d94f5b72f73" (UID: "392fb7ab-dcd4-4b8b-846f-9d94f5b72f73"). InnerVolumeSpecName "kube-api-access-9h55c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:03.083040 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:03.083006 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/392fb7ab-dcd4-4b8b-846f-9d94f5b72f73-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:03.083040 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:03.083032 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h55c\" (UniqueName: \"kubernetes.io/projected/392fb7ab-dcd4-4b8b-846f-9d94f5b72f73-kube-api-access-9h55c\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:03.649160 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:03.649113 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ff2cd1-nfggg" Oct 30 18:53:03.649160 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:03.649149 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09ff2cd1-nfggg" event={"ID":"392fb7ab-dcd4-4b8b-846f-9d94f5b72f73","Type":"ContainerDied","Data":"8fd180f268c9ef1182f8c49739f18d2f94dd935023ce284f1fcabc2d36d8be4f"} Oct 30 18:53:03.649438 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:03.649200 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fd180f268c9ef1182f8c49739f18d2f94dd935023ce284f1fcabc2d36d8be4f" Oct 30 18:53:07.505490 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.505457 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs"] Oct 30 18:53:07.505917 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.505890 4531 cpu_manager.go:414] "containerMap: removing container" podUID="392fb7ab-dcd4-4b8b-846f-9d94f5b72f73" containerName="revoke-keys" Oct 30 18:53:07.505917 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.505902 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="392fb7ab-dcd4-4b8b-846f-9d94f5b72f73" containerName="revoke-keys" Oct 30 18:53:07.505994 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.505977 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="392fb7ab-dcd4-4b8b-846f-9d94f5b72f73" containerName="revoke-keys" Oct 30 18:53:07.509387 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.509371 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.529596 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.529561 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-podinfo\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.529596 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.529593 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0c5609-7aa9-4360-9818-6bdbc744554a-istiod-ca-cert\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.529802 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.529622 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-envoy\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.529802 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.529743 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-token\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.529802 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.529782 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-credential-socket\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.529903 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.529883 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-workload-certs\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.529975 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.529964 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-data\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.530011 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.529995 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-workload-socket\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.530076 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.530060 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79t7k\" (UniqueName: \"kubernetes.io/projected/6f0c5609-7aa9-4360-9818-6bdbc744554a-kube-api-access-79t7k\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.620652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.620615 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-3b6013-openshift-default-dockercfg-4sf9f\"" Oct 30 18:53:07.624370 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.624341 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs"] Oct 30 18:53:07.631459 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.631427 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-credential-socket\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.631622 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.631486 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-workload-certs\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.631622 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.631556 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-data\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.631622 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.631588 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-workload-socket\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.631622 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.631613 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79t7k\" (UniqueName: \"kubernetes.io/projected/6f0c5609-7aa9-4360-9818-6bdbc744554a-kube-api-access-79t7k\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.631859 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.631691 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-podinfo\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.631859 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.631719 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0c5609-7aa9-4360-9818-6bdbc744554a-istiod-ca-cert\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.631859 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.631761 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-envoy\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.631859 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.631805 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-token\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.632056 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.631920 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-credential-socket\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.632113 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.632069 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-workload-socket\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.632467 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.632418 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-data\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.632577 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.632455 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-workload-certs\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.632688 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.632658 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0c5609-7aa9-4360-9818-6bdbc744554a-istiod-ca-cert\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.634396 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.634367 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-envoy\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.634697 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.634672 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-podinfo\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.699843 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.699804 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79t7k\" (UniqueName: \"kubernetes.io/projected/6f0c5609-7aa9-4360-9818-6bdbc744554a-kube-api-access-79t7k\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.701949 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.701922 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-token\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:07.821917 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:07.821823 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:08.290752 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:53:08.290700 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0c5609_7aa9_4360_9818_6bdbc744554a.slice/crio-c58935d044404fc069ac3e82ec867364068e07d064afdafdb0472c9c274f2aad WatchSource:0}: Error finding container c58935d044404fc069ac3e82ec867364068e07d064afdafdb0472c9c274f2aad: Status 404 returned error can't find the container with id c58935d044404fc069ac3e82ec867364068e07d064afdafdb0472c9c274f2aad Oct 30 18:53:08.293337 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:08.293303 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs"] Oct 30 18:53:08.295441 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:08.295391 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:53:08.295565 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:08.295494 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:53:08.295565 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:08.295541 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:53:08.699002 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:08.698906 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" event={"ID":"6f0c5609-7aa9-4360-9818-6bdbc744554a","Type":"ContainerStarted","Data":"e644fefc8000830a5149b6ac0d5718549d3bb1a3e2ded1428c20ebda98406dde"} Oct 30 18:53:08.699002 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:08.698986 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" event={"ID":"6f0c5609-7aa9-4360-9818-6bdbc744554a","Type":"ContainerStarted","Data":"c58935d044404fc069ac3e82ec867364068e07d064afdafdb0472c9c274f2aad"} Oct 30 18:53:08.833155 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:08.824657 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:08.853155 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:08.845484 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" podStartSLOduration=1.845460254 podStartE2EDuration="1.845460254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:08.824567225 +0000 UTC m=+1148.851672187" watchObservedRunningTime="2026-10-30 18:53:08.845460254 +0000 UTC m=+1148.872565208" Oct 30 18:53:09.827574 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:09.827212 4531 patch_prober.go:29] interesting pod/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:53:09.827574 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:09.827294 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" podUID="6f0c5609-7aa9-4360-9818-6bdbc744554a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 18:53:10.540731 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.540683 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d7196a3-fftps"] Oct 30 18:53:10.544735 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.544715 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7196a3-fftps" Oct 30 18:53:10.589587 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.589554 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d7196a3-fftps"] Oct 30 18:53:10.602760 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.602731 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df118305-439d-446d-8175-2f7894fc6302-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d7196a3-fftps\" (UID: \"df118305-439d-446d-8175-2f7894fc6302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7196a3-fftps" Oct 30 18:53:10.602955 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.602779 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df118305-439d-446d-8175-2f7894fc6302-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d7196a3-fftps\" (UID: \"df118305-439d-446d-8175-2f7894fc6302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7196a3-fftps" Oct 30 18:53:10.602955 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.602908 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xxbg\" (UniqueName: \"kubernetes.io/projected/df118305-439d-446d-8175-2f7894fc6302-kube-api-access-6xxbg\") pod \"maas-api-revoke-sub-8d7196a3-fftps\" (UID: \"df118305-439d-446d-8175-2f7894fc6302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7196a3-fftps" Oct 30 18:53:10.647150 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.646518 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:53:10.647150 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.646536 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:53:10.647150 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.646686 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:53:10.647150 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.646774 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:53:10.655542 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.655502 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a4e1823-jg6p8"] Oct 30 18:53:10.659341 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.659318 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4e1823-jg6p8" Oct 30 18:53:10.703961 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.703921 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcz8g\" (UniqueName: \"kubernetes.io/projected/0f9b77fb-c20a-4e51-96be-3e6580a8dc5d-kube-api-access-zcz8g\") pod \"maas-api-revoke-sub-0a4e1823-jg6p8\" (UID: \"0f9b77fb-c20a-4e51-96be-3e6580a8dc5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4e1823-jg6p8" Oct 30 18:53:10.703961 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.703959 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9b77fb-c20a-4e51-96be-3e6580a8dc5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a4e1823-jg6p8\" (UID: \"0f9b77fb-c20a-4e51-96be-3e6580a8dc5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4e1823-jg6p8" Oct 30 18:53:10.704210 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.703999 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df118305-439d-446d-8175-2f7894fc6302-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d7196a3-fftps\" (UID: \"df118305-439d-446d-8175-2f7894fc6302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7196a3-fftps" Oct 30 18:53:10.704210 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.704093 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9b77fb-c20a-4e51-96be-3e6580a8dc5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a4e1823-jg6p8\" (UID: \"0f9b77fb-c20a-4e51-96be-3e6580a8dc5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4e1823-jg6p8" Oct 30 18:53:10.704210 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.704170 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df118305-439d-446d-8175-2f7894fc6302-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d7196a3-fftps\" (UID: \"df118305-439d-446d-8175-2f7894fc6302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7196a3-fftps" Oct 30 18:53:10.704309 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.704220 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xxbg\" (UniqueName: \"kubernetes.io/projected/df118305-439d-446d-8175-2f7894fc6302-kube-api-access-6xxbg\") pod \"maas-api-revoke-sub-8d7196a3-fftps\" (UID: \"df118305-439d-446d-8175-2f7894fc6302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7196a3-fftps" Oct 30 18:53:10.704895 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.704821 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df118305-439d-446d-8175-2f7894fc6302-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d7196a3-fftps\" (UID: \"df118305-439d-446d-8175-2f7894fc6302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7196a3-fftps" Oct 30 18:53:10.706641 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.706619 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df118305-439d-446d-8175-2f7894fc6302-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d7196a3-fftps\" (UID: \"df118305-439d-446d-8175-2f7894fc6302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7196a3-fftps" Oct 30 18:53:10.748888 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.748849 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a4e1823-jg6p8"] Oct 30 18:53:10.766232 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.766189 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xxbg\" (UniqueName: \"kubernetes.io/projected/df118305-439d-446d-8175-2f7894fc6302-kube-api-access-6xxbg\") pod \"maas-api-revoke-sub-8d7196a3-fftps\" (UID: \"df118305-439d-446d-8175-2f7894fc6302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7196a3-fftps" Oct 30 18:53:10.805532 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.805437 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcz8g\" (UniqueName: \"kubernetes.io/projected/0f9b77fb-c20a-4e51-96be-3e6580a8dc5d-kube-api-access-zcz8g\") pod \"maas-api-revoke-sub-0a4e1823-jg6p8\" (UID: \"0f9b77fb-c20a-4e51-96be-3e6580a8dc5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4e1823-jg6p8" Oct 30 18:53:10.805532 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.805482 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9b77fb-c20a-4e51-96be-3e6580a8dc5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a4e1823-jg6p8\" (UID: \"0f9b77fb-c20a-4e51-96be-3e6580a8dc5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4e1823-jg6p8" Oct 30 18:53:10.805774 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.805559 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9b77fb-c20a-4e51-96be-3e6580a8dc5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a4e1823-jg6p8\" (UID: \"0f9b77fb-c20a-4e51-96be-3e6580a8dc5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4e1823-jg6p8" Oct 30 18:53:10.806195 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.806173 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9b77fb-c20a-4e51-96be-3e6580a8dc5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a4e1823-jg6p8\" (UID: \"0f9b77fb-c20a-4e51-96be-3e6580a8dc5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4e1823-jg6p8" Oct 30 18:53:10.808049 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.808032 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9b77fb-c20a-4e51-96be-3e6580a8dc5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a4e1823-jg6p8\" (UID: \"0f9b77fb-c20a-4e51-96be-3e6580a8dc5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4e1823-jg6p8" Oct 30 18:53:10.841676 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.841637 4531 patch_prober.go:29] interesting pod/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:53:10.842202 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.841689 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" podUID="6f0c5609-7aa9-4360-9818-6bdbc744554a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 18:53:10.858238 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.858205 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7196a3-fftps" Oct 30 18:53:10.867080 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.867056 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcz8g\" (UniqueName: \"kubernetes.io/projected/0f9b77fb-c20a-4e51-96be-3e6580a8dc5d-kube-api-access-zcz8g\") pod \"maas-api-revoke-sub-0a4e1823-jg6p8\" (UID: \"0f9b77fb-c20a-4e51-96be-3e6580a8dc5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4e1823-jg6p8" Oct 30 18:53:10.973235 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:10.972923 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4e1823-jg6p8" Oct 30 18:53:11.255271 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:11.255248 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d7196a3-fftps"] Oct 30 18:53:11.256272 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:53:11.256243 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf118305_439d_446d_8175_2f7894fc6302.slice/crio-55aa15caedcaf3d9ce154162576e595bfd9b7878a9509f37b45c8d8e2e82d200 WatchSource:0}: Error finding container 55aa15caedcaf3d9ce154162576e595bfd9b7878a9509f37b45c8d8e2e82d200: Status 404 returned error can't find the container with id 55aa15caedcaf3d9ce154162576e595bfd9b7878a9509f37b45c8d8e2e82d200 Oct 30 18:53:11.385497 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:11.385468 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a4e1823-jg6p8"] Oct 30 18:53:11.386947 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:53:11.386921 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9b77fb_c20a_4e51_96be_3e6580a8dc5d.slice/crio-46027f75b3c9d17042f3d8a736c43d6026ff08297cbf2a4dc9f2259a6146798c WatchSource:0}: Error finding container 46027f75b3c9d17042f3d8a736c43d6026ff08297cbf2a4dc9f2259a6146798c: Status 404 returned error can't find the container with id 46027f75b3c9d17042f3d8a736c43d6026ff08297cbf2a4dc9f2259a6146798c Oct 30 18:53:11.745715 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:11.745682 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4e1823-jg6p8" event={"ID":"0f9b77fb-c20a-4e51-96be-3e6580a8dc5d","Type":"ContainerStarted","Data":"46027f75b3c9d17042f3d8a736c43d6026ff08297cbf2a4dc9f2259a6146798c"} Oct 30 18:53:11.747459 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:11.747429 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7196a3-fftps" event={"ID":"df118305-439d-446d-8175-2f7894fc6302","Type":"ContainerStarted","Data":"dd65f2c9f304b054cfd539a3692a982516a96015b529784ea7e1f419f7be3309"} Oct 30 18:53:11.747611 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:11.747478 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7196a3-fftps" event={"ID":"df118305-439d-446d-8175-2f7894fc6302","Type":"ContainerStarted","Data":"55aa15caedcaf3d9ce154162576e595bfd9b7878a9509f37b45c8d8e2e82d200"} Oct 30 18:53:11.842582 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:11.842532 4531 patch_prober.go:29] interesting pod/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:53:11.843086 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:11.842599 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" podUID="6f0c5609-7aa9-4360-9818-6bdbc744554a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 18:53:11.866723 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:11.866663 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7196a3-fftps" podStartSLOduration=1.8666456070000002 podStartE2EDuration="1.866645607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:11.863676149 +0000 UTC m=+1151.890781107" watchObservedRunningTime="2026-10-30 18:53:11.866645607 +0000 UTC m=+1151.893750566" Oct 30 18:53:12.142089 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.142021 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" podUID="5686e156-9f19-401b-b305-99acd65a6ce5" containerName="main" containerID="cri-o://e872f062eb13641426f14160fe714988385615574e6bd8fb5e02ea87c1d1cb32" gracePeriod=45 Oct 30 18:53:12.147964 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.147932 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" podUID="5686e156-9f19-401b-b305-99acd65a6ce5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": dial tcp 10.133.0.54:8000: connect: connection refused" Oct 30 18:53:12.498275 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.498244 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" Oct 30 18:53:12.652706 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.652628 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-model-cache\") pod \"5686e156-9f19-401b-b305-99acd65a6ce5\" (UID: \"5686e156-9f19-401b-b305-99acd65a6ce5\") " Oct 30 18:53:12.652706 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.652685 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-home\" (UniqueName: \"kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-home\") pod \"5686e156-9f19-401b-b305-99acd65a6ce5\" (UID: \"5686e156-9f19-401b-b305-99acd65a6ce5\") " Oct 30 18:53:12.652952 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.652744 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5686e156-9f19-401b-b305-99acd65a6ce5-kube-api-access-lt45h\" (UniqueName: \"kubernetes.io/projected/5686e156-9f19-401b-b305-99acd65a6ce5-kube-api-access-lt45h\") pod \"5686e156-9f19-401b-b305-99acd65a6ce5\" (UID: \"5686e156-9f19-401b-b305-99acd65a6ce5\") " Oct 30 18:53:12.652952 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.652839 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5686e156-9f19-401b-b305-99acd65a6ce5-tls-certs\" (UniqueName: \"kubernetes.io/secret/5686e156-9f19-401b-b305-99acd65a6ce5-tls-certs\") pod \"5686e156-9f19-401b-b305-99acd65a6ce5\" (UID: \"5686e156-9f19-401b-b305-99acd65a6ce5\") " Oct 30 18:53:12.652952 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.652870 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-tmp-dir\") pod \"5686e156-9f19-401b-b305-99acd65a6ce5\" (UID: \"5686e156-9f19-401b-b305-99acd65a6ce5\") " Oct 30 18:53:12.652952 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.652916 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-dshm\") pod \"5686e156-9f19-401b-b305-99acd65a6ce5\" (UID: \"5686e156-9f19-401b-b305-99acd65a6ce5\") " Oct 30 18:53:12.653480 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.653416 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-tmp-dir" pod "5686e156-9f19-401b-b305-99acd65a6ce5" (UID: "5686e156-9f19-401b-b305-99acd65a6ce5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:53:12.653480 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.653454 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-home" pod "5686e156-9f19-401b-b305-99acd65a6ce5" (UID: "5686e156-9f19-401b-b305-99acd65a6ce5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:53:12.653652 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.653564 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-model-cache" pod "5686e156-9f19-401b-b305-99acd65a6ce5" (UID: "5686e156-9f19-401b-b305-99acd65a6ce5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:53:12.655751 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.655724 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-dshm" pod "5686e156-9f19-401b-b305-99acd65a6ce5" (UID: "5686e156-9f19-401b-b305-99acd65a6ce5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:53:12.655857 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.655788 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5686e156-9f19-401b-b305-99acd65a6ce5-kube-api-access-lt45h" pod "5686e156-9f19-401b-b305-99acd65a6ce5" (UID: "5686e156-9f19-401b-b305-99acd65a6ce5"). InnerVolumeSpecName "kube-api-access-lt45h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:12.655936 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.655909 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5686e156-9f19-401b-b305-99acd65a6ce5-tls-certs" pod "5686e156-9f19-401b-b305-99acd65a6ce5" (UID: "5686e156-9f19-401b-b305-99acd65a6ce5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:12.753912 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.753880 4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5686e156-9f19-401b-b305-99acd65a6ce5-tls-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:12.753912 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.753910 4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-tmp-dir\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:12.753912 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.753923 4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-dshm\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:12.754164 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.753936 4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-model-cache\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:12.754164 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.753948 4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5686e156-9f19-401b-b305-99acd65a6ce5-home\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:12.754164 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.753960 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt45h\" (UniqueName: \"kubernetes.io/projected/5686e156-9f19-401b-b305-99acd65a6ce5-kube-api-access-lt45h\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:12.754722 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.754698 4531 generic.go:356] "Generic (PLEG): container finished" podID="e20d0816-9f25-4e7f-bf65-56a30e7a05eb" containerID="224c24b30098c77267c37b5512244d9afdcebc2bb36ee94e796258adf5671053" exitCode=0 Oct 30 18:53:12.754805 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.754779 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tfmtb-7bqnn" event={"ID":"e20d0816-9f25-4e7f-bf65-56a30e7a05eb","Type":"ContainerDied","Data":"224c24b30098c77267c37b5512244d9afdcebc2bb36ee94e796258adf5671053"} Oct 30 18:53:12.756032 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.756011 4531 generic.go:356] "Generic (PLEG): container finished" podID="5686e156-9f19-401b-b305-99acd65a6ce5" containerID="e872f062eb13641426f14160fe714988385615574e6bd8fb5e02ea87c1d1cb32" exitCode=0 Oct 30 18:53:12.756189 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.756058 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" Oct 30 18:53:12.756189 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.756071 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" event={"ID":"5686e156-9f19-401b-b305-99acd65a6ce5","Type":"ContainerDied","Data":"e872f062eb13641426f14160fe714988385615574e6bd8fb5e02ea87c1d1cb32"} Oct 30 18:53:12.756189 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.756108 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x" event={"ID":"5686e156-9f19-401b-b305-99acd65a6ce5","Type":"ContainerDied","Data":"e07d225956abfb17462dde0f72f5f838f98c95f2c5bd5896cfb3ffc10ddfaf2b"} Oct 30 18:53:12.756189 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.756141 4531 scope.go:122] "RemoveContainer" containerID="e872f062eb13641426f14160fe714988385615574e6bd8fb5e02ea87c1d1cb32" Oct 30 18:53:12.757784 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.757692 4531 generic.go:356] "Generic (PLEG): container finished" podID="0f9b77fb-c20a-4e51-96be-3e6580a8dc5d" containerID="9f7e62877d38381f454294d497b4433a0b02099ce0256409e5674ac1ffaad050" exitCode=0 Oct 30 18:53:12.757784 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.757728 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4e1823-jg6p8" event={"ID":"0f9b77fb-c20a-4e51-96be-3e6580a8dc5d","Type":"ContainerDied","Data":"9f7e62877d38381f454294d497b4433a0b02099ce0256409e5674ac1ffaad050"} Oct 30 18:53:12.759553 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.759532 4531 generic.go:356] "Generic (PLEG): container finished" podID="df118305-439d-446d-8175-2f7894fc6302" containerID="dd65f2c9f304b054cfd539a3692a982516a96015b529784ea7e1f419f7be3309" exitCode=0 Oct 30 18:53:12.759663 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.759632 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7196a3-fftps" event={"ID":"df118305-439d-446d-8175-2f7894fc6302","Type":"ContainerDied","Data":"dd65f2c9f304b054cfd539a3692a982516a96015b529784ea7e1f419f7be3309"} Oct 30 18:53:12.767202 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.767182 4531 scope.go:122] "RemoveContainer" containerID="e872f062eb13641426f14160fe714988385615574e6bd8fb5e02ea87c1d1cb32" Oct 30 18:53:12.767535 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:53:12.767512 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e872f062eb13641426f14160fe714988385615574e6bd8fb5e02ea87c1d1cb32\": container with ID starting with e872f062eb13641426f14160fe714988385615574e6bd8fb5e02ea87c1d1cb32 not found: ID does not exist" containerID="e872f062eb13641426f14160fe714988385615574e6bd8fb5e02ea87c1d1cb32" Oct 30 18:53:12.767601 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.767544 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e872f062eb13641426f14160fe714988385615574e6bd8fb5e02ea87c1d1cb32"} err="failed to get container status \"e872f062eb13641426f14160fe714988385615574e6bd8fb5e02ea87c1d1cb32\": rpc error: code = NotFound desc = could not find container \"e872f062eb13641426f14160fe714988385615574e6bd8fb5e02ea87c1d1cb32\": container with ID starting with e872f062eb13641426f14160fe714988385615574e6bd8fb5e02ea87c1d1cb32 not found: ID does not exist" Oct 30 18:53:12.823054 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.822598 4531 patch_prober.go:29] interesting pod/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:53:12.823054 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.822662 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" podUID="6f0c5609-7aa9-4360-9818-6bdbc744554a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 18:53:12.982638 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.982540 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x"] Oct 30 18:53:12.986854 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:12.986823 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/e2e-conflict-b-d331d324-kserve-769f6c6fcf-85f4x"] Oct 30 18:53:13.829188 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:13.824541 4531 patch_prober.go:29] interesting pod/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:53:13.829188 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:13.824598 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" podUID="6f0c5609-7aa9-4360-9818-6bdbc744554a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 18:53:13.978659 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:13.978505 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4e1823-jg6p8" Oct 30 18:53:14.059630 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.059601 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tfmtb-7bqnn" Oct 30 18:53:14.062876 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.062855 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7196a3-fftps" Oct 30 18:53:14.069592 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.069556 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f9b77fb-c20a-4e51-96be-3e6580a8dc5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9b77fb-c20a-4e51-96be-3e6580a8dc5d-maas-api-xks-ca\") pod \"0f9b77fb-c20a-4e51-96be-3e6580a8dc5d\" (UID: \"0f9b77fb-c20a-4e51-96be-3e6580a8dc5d\") " Oct 30 18:53:14.069747 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.069669 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f9b77fb-c20a-4e51-96be-3e6580a8dc5d-kube-api-access-zcz8g\" (UniqueName: \"kubernetes.io/projected/0f9b77fb-c20a-4e51-96be-3e6580a8dc5d-kube-api-access-zcz8g\") pod \"0f9b77fb-c20a-4e51-96be-3e6580a8dc5d\" (UID: \"0f9b77fb-c20a-4e51-96be-3e6580a8dc5d\") " Oct 30 18:53:14.069747 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.069741 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f9b77fb-c20a-4e51-96be-3e6580a8dc5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9b77fb-c20a-4e51-96be-3e6580a8dc5d-maas-api-openshift-ca\") pod \"0f9b77fb-c20a-4e51-96be-3e6580a8dc5d\" (UID: \"0f9b77fb-c20a-4e51-96be-3e6580a8dc5d\") " Oct 30 18:53:14.070678 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.070654 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9b77fb-c20a-4e51-96be-3e6580a8dc5d-maas-api-openshift-ca" pod "0f9b77fb-c20a-4e51-96be-3e6580a8dc5d" (UID: "0f9b77fb-c20a-4e51-96be-3e6580a8dc5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:14.072348 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.072326 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f9b77fb-c20a-4e51-96be-3e6580a8dc5d-maas-api-xks-ca" pod "0f9b77fb-c20a-4e51-96be-3e6580a8dc5d" (UID: "0f9b77fb-c20a-4e51-96be-3e6580a8dc5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:14.072745 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.072730 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9b77fb-c20a-4e51-96be-3e6580a8dc5d-kube-api-access-zcz8g" pod "0f9b77fb-c20a-4e51-96be-3e6580a8dc5d" (UID: "0f9b77fb-c20a-4e51-96be-3e6580a8dc5d"). InnerVolumeSpecName "kube-api-access-zcz8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:14.102948 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.102905 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5686e156-9f19-401b-b305-99acd65a6ce5" path="/var/lib/kubelet/pods/5686e156-9f19-401b-b305-99acd65a6ce5/volumes" Oct 30 18:53:14.170495 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.170454 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-kube-api-access-x6f4t\" (UniqueName: \"kubernetes.io/projected/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-kube-api-access-x6f4t\") pod \"e20d0816-9f25-4e7f-bf65-56a30e7a05eb\" (UID: \"e20d0816-9f25-4e7f-bf65-56a30e7a05eb\") " Oct 30 18:53:14.170495 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.170495 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-service-ca-bundle\") pod \"e20d0816-9f25-4e7f-bf65-56a30e7a05eb\" (UID: \"e20d0816-9f25-4e7f-bf65-56a30e7a05eb\") " Oct 30 18:53:14.170749 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.170543 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df118305-439d-446d-8175-2f7894fc6302-kube-api-access-6xxbg\" (UniqueName: \"kubernetes.io/projected/df118305-439d-446d-8175-2f7894fc6302-kube-api-access-6xxbg\") pod \"df118305-439d-446d-8175-2f7894fc6302\" (UID: \"df118305-439d-446d-8175-2f7894fc6302\") " Oct 30 18:53:14.170749 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.170562 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-trusted-ca-bundle\") pod \"e20d0816-9f25-4e7f-bf65-56a30e7a05eb\" (UID: \"e20d0816-9f25-4e7f-bf65-56a30e7a05eb\") " Oct 30 18:53:14.170749 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.170602 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df118305-439d-446d-8175-2f7894fc6302-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df118305-439d-446d-8175-2f7894fc6302-maas-api-xks-ca\") pod \"df118305-439d-446d-8175-2f7894fc6302\" (UID: \"df118305-439d-446d-8175-2f7894fc6302\") " Oct 30 18:53:14.170749 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.170623 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-archives-path\") pod \"e20d0816-9f25-4e7f-bf65-56a30e7a05eb\" (UID: \"e20d0816-9f25-4e7f-bf65-56a30e7a05eb\") " Oct 30 18:53:14.170749 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.170661 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df118305-439d-446d-8175-2f7894fc6302-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df118305-439d-446d-8175-2f7894fc6302-maas-api-openshift-ca\") pod \"df118305-439d-446d-8175-2f7894fc6302\" (UID: \"df118305-439d-446d-8175-2f7894fc6302\") " Oct 30 18:53:14.171376 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.171217 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9b77fb-c20a-4e51-96be-3e6580a8dc5d-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:14.171473 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.171422 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9b77fb-c20a-4e51-96be-3e6580a8dc5d-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:14.171473 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.171441 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcz8g\" (UniqueName: \"kubernetes.io/projected/0f9b77fb-c20a-4e51-96be-3e6580a8dc5d-kube-api-access-zcz8g\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:14.171473 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.171325 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-trusted-ca-bundle" pod "e20d0816-9f25-4e7f-bf65-56a30e7a05eb" (UID: "e20d0816-9f25-4e7f-bf65-56a30e7a05eb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:14.171473 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.171438 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-service-ca-bundle" pod "e20d0816-9f25-4e7f-bf65-56a30e7a05eb" (UID: "e20d0816-9f25-4e7f-bf65-56a30e7a05eb"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:14.171683 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.171639 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-archives-path" pod "e20d0816-9f25-4e7f-bf65-56a30e7a05eb" (UID: "e20d0816-9f25-4e7f-bf65-56a30e7a05eb"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:53:14.171939 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.171915 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df118305-439d-446d-8175-2f7894fc6302-maas-api-openshift-ca" pod "df118305-439d-446d-8175-2f7894fc6302" (UID: "df118305-439d-446d-8175-2f7894fc6302"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:14.173283 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.173201 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-kube-api-access-x6f4t" pod "e20d0816-9f25-4e7f-bf65-56a30e7a05eb" (UID: "e20d0816-9f25-4e7f-bf65-56a30e7a05eb"). InnerVolumeSpecName "kube-api-access-x6f4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:14.174602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.174569 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df118305-439d-446d-8175-2f7894fc6302-maas-api-xks-ca" pod "df118305-439d-446d-8175-2f7894fc6302" (UID: "df118305-439d-446d-8175-2f7894fc6302"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:14.174817 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.174798 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df118305-439d-446d-8175-2f7894fc6302-kube-api-access-6xxbg" pod "df118305-439d-446d-8175-2f7894fc6302" (UID: "df118305-439d-446d-8175-2f7894fc6302"). InnerVolumeSpecName "kube-api-access-6xxbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:14.293663 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.293420 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6f4t\" (UniqueName: \"kubernetes.io/projected/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-kube-api-access-x6f4t\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:14.293868 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.293852 4531 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-service-ca-bundle\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:14.293932 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.293924 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xxbg\" (UniqueName: \"kubernetes.io/projected/df118305-439d-446d-8175-2f7894fc6302-kube-api-access-6xxbg\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:14.293981 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.293974 4531 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-trusted-ca-bundle\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:14.294036 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.294027 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df118305-439d-446d-8175-2f7894fc6302-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:14.294087 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.294080 4531 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e20d0816-9f25-4e7f-bf65-56a30e7a05eb-archives-path\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:14.294159 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.294148 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df118305-439d-446d-8175-2f7894fc6302-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:14.773258 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.773218 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tfmtb-7bqnn" Oct 30 18:53:14.773469 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.773259 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tfmtb-7bqnn" event={"ID":"e20d0816-9f25-4e7f-bf65-56a30e7a05eb","Type":"ContainerDied","Data":"26568c18b600e86b12cdfb1c3907447817195e1889c0d7b0af879fd35a53e84b"} Oct 30 18:53:14.773469 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.773311 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26568c18b600e86b12cdfb1c3907447817195e1889c0d7b0af879fd35a53e84b" Oct 30 18:53:14.774938 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.774905 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4e1823-jg6p8" event={"ID":"0f9b77fb-c20a-4e51-96be-3e6580a8dc5d","Type":"ContainerDied","Data":"46027f75b3c9d17042f3d8a736c43d6026ff08297cbf2a4dc9f2259a6146798c"} Oct 30 18:53:14.775091 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.774958 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46027f75b3c9d17042f3d8a736c43d6026ff08297cbf2a4dc9f2259a6146798c" Oct 30 18:53:14.775091 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.774916 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4e1823-jg6p8" Oct 30 18:53:14.776422 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.776404 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7196a3-fftps" Oct 30 18:53:14.776532 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.776455 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7196a3-fftps" event={"ID":"df118305-439d-446d-8175-2f7894fc6302","Type":"ContainerDied","Data":"55aa15caedcaf3d9ce154162576e595bfd9b7878a9509f37b45c8d8e2e82d200"} Oct 30 18:53:14.776532 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.776497 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55aa15caedcaf3d9ce154162576e595bfd9b7878a9509f37b45c8d8e2e82d200" Oct 30 18:53:14.822778 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.822746 4531 patch_prober.go:29] interesting pod/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:53:14.822920 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:14.822797 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" podUID="6f0c5609-7aa9-4360-9818-6bdbc744554a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 18:53:15.827174 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:15.827123 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:15.827657 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:15.827533 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:15.828189 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:15.828171 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 18:53:26.407712 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:53:26.407663 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:53:27.132342 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.132306 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf4f477c-5c9jv"] Oct 30 18:53:27.132699 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.132687 4531 cpu_manager.go:414] "containerMap: removing container" podUID="0f9b77fb-c20a-4e51-96be-3e6580a8dc5d" containerName="revoke-keys" Oct 30 18:53:27.132742 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.132699 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9b77fb-c20a-4e51-96be-3e6580a8dc5d" containerName="revoke-keys" Oct 30 18:53:27.132742 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.132715 4531 cpu_manager.go:414] "containerMap: removing container" podUID="5686e156-9f19-401b-b305-99acd65a6ce5" containerName="main" Oct 30 18:53:27.132742 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.132720 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5686e156-9f19-401b-b305-99acd65a6ce5" containerName="main" Oct 30 18:53:27.132742 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.132735 4531 cpu_manager.go:414] "containerMap: removing container" podUID="df118305-439d-446d-8175-2f7894fc6302" containerName="revoke-keys" Oct 30 18:53:27.132742 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.132741 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df118305-439d-446d-8175-2f7894fc6302" containerName="revoke-keys" Oct 30 18:53:27.132893 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.132760 4531 cpu_manager.go:414] "containerMap: removing container" podUID="e20d0816-9f25-4e7f-bf65-56a30e7a05eb" containerName="insights-gathering" Oct 30 18:53:27.132893 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.132765 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e20d0816-9f25-4e7f-bf65-56a30e7a05eb" containerName="insights-gathering" Oct 30 18:53:27.132893 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.132828 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5686e156-9f19-401b-b305-99acd65a6ce5" containerName="main" Oct 30 18:53:27.132893 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.132838 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="df118305-439d-446d-8175-2f7894fc6302" containerName="revoke-keys" Oct 30 18:53:27.132893 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.132847 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e20d0816-9f25-4e7f-bf65-56a30e7a05eb" containerName="insights-gathering" Oct 30 18:53:27.132893 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.132853 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9b77fb-c20a-4e51-96be-3e6580a8dc5d" containerName="revoke-keys" Oct 30 18:53:27.135979 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.135959 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4f477c-5c9jv" Oct 30 18:53:27.188071 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.188045 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:53:27.224032 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.223989 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd8446ae-527e-451f-81f6-88f7afd3f457-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf4f477c-5c9jv\" (UID: \"bd8446ae-527e-451f-81f6-88f7afd3f457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4f477c-5c9jv" Oct 30 18:53:27.224236 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.224039 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd8446ae-527e-451f-81f6-88f7afd3f457-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf4f477c-5c9jv\" (UID: \"bd8446ae-527e-451f-81f6-88f7afd3f457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4f477c-5c9jv" Oct 30 18:53:27.224307 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.224282 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhkbd\" (UniqueName: \"kubernetes.io/projected/bd8446ae-527e-451f-81f6-88f7afd3f457-kube-api-access-fhkbd\") pod \"maas-api-revoke-sub-bf4f477c-5c9jv\" (UID: \"bd8446ae-527e-451f-81f6-88f7afd3f457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4f477c-5c9jv" Oct 30 18:53:27.237345 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.237314 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:53:27.237475 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.237378 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:53:27.237514 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.237486 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:53:27.240169 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.240149 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf4f477c-5c9jv"] Oct 30 18:53:27.325422 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.325387 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd8446ae-527e-451f-81f6-88f7afd3f457-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf4f477c-5c9jv\" (UID: \"bd8446ae-527e-451f-81f6-88f7afd3f457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4f477c-5c9jv" Oct 30 18:53:27.325422 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.325423 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd8446ae-527e-451f-81f6-88f7afd3f457-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf4f477c-5c9jv\" (UID: \"bd8446ae-527e-451f-81f6-88f7afd3f457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4f477c-5c9jv" Oct 30 18:53:27.325657 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.325498 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhkbd\" (UniqueName: \"kubernetes.io/projected/bd8446ae-527e-451f-81f6-88f7afd3f457-kube-api-access-fhkbd\") pod \"maas-api-revoke-sub-bf4f477c-5c9jv\" (UID: \"bd8446ae-527e-451f-81f6-88f7afd3f457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4f477c-5c9jv" Oct 30 18:53:27.326257 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.326194 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd8446ae-527e-451f-81f6-88f7afd3f457-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf4f477c-5c9jv\" (UID: \"bd8446ae-527e-451f-81f6-88f7afd3f457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4f477c-5c9jv" Oct 30 18:53:27.327954 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.327933 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd8446ae-527e-451f-81f6-88f7afd3f457-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf4f477c-5c9jv\" (UID: \"bd8446ae-527e-451f-81f6-88f7afd3f457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4f477c-5c9jv" Oct 30 18:53:27.387020 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.386945 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhkbd\" (UniqueName: \"kubernetes.io/projected/bd8446ae-527e-451f-81f6-88f7afd3f457-kube-api-access-fhkbd\") pod \"maas-api-revoke-sub-bf4f477c-5c9jv\" (UID: \"bd8446ae-527e-451f-81f6-88f7afd3f457\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4f477c-5c9jv" Oct 30 18:53:27.448099 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.448062 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4f477c-5c9jv" Oct 30 18:53:27.585647 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.585610 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b161d939-k5xqz"] Oct 30 18:53:27.589500 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.589481 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b161d939-k5xqz" Oct 30 18:53:27.627847 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.627821 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae1ac14-8378-4501-ba25-4e59f1e674ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b161d939-k5xqz\" (UID: \"5ae1ac14-8378-4501-ba25-4e59f1e674ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b161d939-k5xqz" Oct 30 18:53:27.628005 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.627867 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae1ac14-8378-4501-ba25-4e59f1e674ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b161d939-k5xqz\" (UID: \"5ae1ac14-8378-4501-ba25-4e59f1e674ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b161d939-k5xqz" Oct 30 18:53:27.628046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.627996 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clw96\" (UniqueName: \"kubernetes.io/projected/5ae1ac14-8378-4501-ba25-4e59f1e674ac-kube-api-access-clw96\") pod \"maas-api-revoke-sub-b161d939-k5xqz\" (UID: \"5ae1ac14-8378-4501-ba25-4e59f1e674ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b161d939-k5xqz" Oct 30 18:53:27.648093 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.648002 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b161d939-k5xqz"] Oct 30 18:53:27.728919 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.728886 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clw96\" (UniqueName: \"kubernetes.io/projected/5ae1ac14-8378-4501-ba25-4e59f1e674ac-kube-api-access-clw96\") pod \"maas-api-revoke-sub-b161d939-k5xqz\" (UID: \"5ae1ac14-8378-4501-ba25-4e59f1e674ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b161d939-k5xqz" Oct 30 18:53:27.729143 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.728962 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae1ac14-8378-4501-ba25-4e59f1e674ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b161d939-k5xqz\" (UID: \"5ae1ac14-8378-4501-ba25-4e59f1e674ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b161d939-k5xqz" Oct 30 18:53:27.729143 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.728998 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae1ac14-8378-4501-ba25-4e59f1e674ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b161d939-k5xqz\" (UID: \"5ae1ac14-8378-4501-ba25-4e59f1e674ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b161d939-k5xqz" Oct 30 18:53:27.729606 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.729586 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae1ac14-8378-4501-ba25-4e59f1e674ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b161d939-k5xqz\" (UID: \"5ae1ac14-8378-4501-ba25-4e59f1e674ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b161d939-k5xqz" Oct 30 18:53:27.731516 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.731492 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae1ac14-8378-4501-ba25-4e59f1e674ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b161d939-k5xqz\" (UID: \"5ae1ac14-8378-4501-ba25-4e59f1e674ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b161d939-k5xqz" Oct 30 18:53:27.789818 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.789782 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clw96\" (UniqueName: \"kubernetes.io/projected/5ae1ac14-8378-4501-ba25-4e59f1e674ac-kube-api-access-clw96\") pod \"maas-api-revoke-sub-b161d939-k5xqz\" (UID: \"5ae1ac14-8378-4501-ba25-4e59f1e674ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b161d939-k5xqz" Oct 30 18:53:27.823574 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.823548 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf4f477c-5c9jv"] Oct 30 18:53:27.824963 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:53:27.824938 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8446ae_527e_451f_81f6_88f7afd3f457.slice/crio-7f2e54f692c766a6e7347925d619fd46f94b6d2720b51e941bedd5ccf613bf92 WatchSource:0}: Error finding container 7f2e54f692c766a6e7347925d619fd46f94b6d2720b51e941bedd5ccf613bf92: Status 404 returned error can't find the container with id 7f2e54f692c766a6e7347925d619fd46f94b6d2720b51e941bedd5ccf613bf92 Oct 30 18:53:27.860188 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.860155 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4f477c-5c9jv" event={"ID":"bd8446ae-527e-451f-81f6-88f7afd3f457","Type":"ContainerStarted","Data":"7f2e54f692c766a6e7347925d619fd46f94b6d2720b51e941bedd5ccf613bf92"} Oct 30 18:53:27.902695 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:27.902611 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b161d939-k5xqz" Oct 30 18:53:28.282480 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.282444 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b161d939-k5xqz"] Oct 30 18:53:28.284561 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:53:28.284532 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae1ac14_8378_4501_ba25_4e59f1e674ac.slice/crio-88d2a5f3b37fb39c50430095c4106335433fc21e23dc97afeca06981ac34e05b WatchSource:0}: Error finding container 88d2a5f3b37fb39c50430095c4106335433fc21e23dc97afeca06981ac34e05b: Status 404 returned error can't find the container with id 88d2a5f3b37fb39c50430095c4106335433fc21e23dc97afeca06981ac34e05b Oct 30 18:53:28.285785 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.285765 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265"] Oct 30 18:53:28.289634 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.289619 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.335448 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.335412 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-envoy\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.335585 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.335461 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-podinfo\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.335585 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.335559 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istiod-ca-cert\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.335707 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.335618 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wxck\" (UniqueName: \"kubernetes.io/projected/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-kube-api-access-5wxck\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.335707 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.335679 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-credential-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.335784 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.335760 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-token\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.335837 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.335820 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-data\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.335903 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.335885 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-workload-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.335961 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.335947 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-workload-certs\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.437518 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.437494 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-credential-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.437629 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.437550 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-token\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.437629 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.437588 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-data\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.437629 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.437611 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-workload-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.437798 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.437658 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-workload-certs\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.437798 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.437689 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-envoy\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.437798 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.437714 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-podinfo\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.437798 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.437751 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istiod-ca-cert\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.437798 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.437782 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wxck\" (UniqueName: \"kubernetes.io/projected/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-kube-api-access-5wxck\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.438055 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.437897 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-credential-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.438055 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.437971 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-data\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.438195 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.438055 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-workload-certs\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.438195 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.438108 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-workload-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.438515 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.438493 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istiod-ca-cert\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.440406 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.440379 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-envoy\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.440514 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.440435 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-podinfo\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.495036 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.494999 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265"] Oct 30 18:53:28.499406 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.499372 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-token\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.500117 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.500094 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wxck\" (UniqueName: \"kubernetes.io/projected/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-kube-api-access-5wxck\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.603631 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.603533 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:28.868043 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.867956 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b161d939-k5xqz" event={"ID":"5ae1ac14-8378-4501-ba25-4e59f1e674ac","Type":"ContainerStarted","Data":"35765236e48f3f92d7803b5c7c09845fc344c5b4d414b7f68cffd4b79a197dfd"} Oct 30 18:53:28.868242 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.868048 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b161d939-k5xqz" event={"ID":"5ae1ac14-8378-4501-ba25-4e59f1e674ac","Type":"ContainerStarted","Data":"88d2a5f3b37fb39c50430095c4106335433fc21e23dc97afeca06981ac34e05b"} Oct 30 18:53:28.869702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.869675 4531 generic.go:356] "Generic (PLEG): container finished" podID="bd8446ae-527e-451f-81f6-88f7afd3f457" containerID="46ea4dfe797ce69a0b8bd1942f28dc408500a47dedc45611102ae78117a3b77b" exitCode=0 Oct 30 18:53:28.869818 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.869712 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4f477c-5c9jv" event={"ID":"bd8446ae-527e-451f-81f6-88f7afd3f457","Type":"ContainerDied","Data":"46ea4dfe797ce69a0b8bd1942f28dc408500a47dedc45611102ae78117a3b77b"} Oct 30 18:53:28.986374 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:28.986320 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b161d939-k5xqz" podStartSLOduration=1.98630201 podStartE2EDuration="1.98630201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:28.983832054 +0000 UTC m=+1169.010937015" watchObservedRunningTime="2026-10-30 18:53:28.98630201 +0000 UTC m=+1169.013406970" Oct 30 18:53:29.036269 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:53:29.036239 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56bbae65_edbd_48ad_b1f6_7c5da8d2d53d.slice/crio-dd4e0045b9d082b38c88cc06699c4f49f2128153dcb24dea0e646aebec02b78b WatchSource:0}: Error finding container dd4e0045b9d082b38c88cc06699c4f49f2128153dcb24dea0e646aebec02b78b: Status 404 returned error can't find the container with id dd4e0045b9d082b38c88cc06699c4f49f2128153dcb24dea0e646aebec02b78b Oct 30 18:53:29.036987 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.036962 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265"] Oct 30 18:53:29.038704 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.038673 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:53:29.038760 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.038737 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:53:29.038804 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.038776 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:53:29.741157 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.740835 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2"] Oct 30 18:53:29.745651 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.745612 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:29.847977 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.847834 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2"] Oct 30 18:53:29.853414 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.853380 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-envoy\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:29.853601 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.853441 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-data\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:29.853601 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.853484 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-workload-certs\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:29.853751 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.853711 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-credential-socket\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:29.853807 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.853754 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-podinfo\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:29.853939 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.853916 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-workload-socket\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:29.854013 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.853971 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efe5fedb-d8ee-418b-89d6-1d1891e436df-istiod-ca-cert\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:29.854072 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.854017 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xfrx\" (UniqueName: \"kubernetes.io/projected/efe5fedb-d8ee-418b-89d6-1d1891e436df-kube-api-access-4xfrx\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:29.854172 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.854071 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-token\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:29.878730 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.877110 4531 generic.go:356] "Generic (PLEG): container finished" podID="5ae1ac14-8378-4501-ba25-4e59f1e674ac" containerID="35765236e48f3f92d7803b5c7c09845fc344c5b4d414b7f68cffd4b79a197dfd" exitCode=0 Oct 30 18:53:29.878730 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.877199 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b161d939-k5xqz" event={"ID":"5ae1ac14-8378-4501-ba25-4e59f1e674ac","Type":"ContainerDied","Data":"35765236e48f3f92d7803b5c7c09845fc344c5b4d414b7f68cffd4b79a197dfd"} Oct 30 18:53:29.879296 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.879270 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" event={"ID":"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d","Type":"ContainerStarted","Data":"305ec0fdca540995d5b873ff12e609e82d95ccb2bf3d18d015d7a60c2e8d95ef"} Oct 30 18:53:29.879361 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.879320 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" event={"ID":"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d","Type":"ContainerStarted","Data":"dd4e0045b9d082b38c88cc06699c4f49f2128153dcb24dea0e646aebec02b78b"} Oct 30 18:53:29.954996 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.954955 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-envoy\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:29.955229 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.955017 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-data\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:29.955229 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.955066 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-workload-certs\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:29.955229 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.955199 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-credential-socket\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:29.955426 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.955233 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-podinfo\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:29.955426 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.955313 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-workload-socket\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:29.955426 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.955338 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efe5fedb-d8ee-418b-89d6-1d1891e436df-istiod-ca-cert\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:29.955426 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.955393 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xfrx\" (UniqueName: \"kubernetes.io/projected/efe5fedb-d8ee-418b-89d6-1d1891e436df-kube-api-access-4xfrx\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:29.955622 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.955452 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-token\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:29.955622 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.955537 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-data\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:29.956361 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.956007 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-workload-certs\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:29.956361 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.956322 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-credential-socket\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:29.956578 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.956553 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-workload-socket\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:29.957063 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.957033 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efe5fedb-d8ee-418b-89d6-1d1891e436df-istiod-ca-cert\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:29.960383 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.959121 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-envoy\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:29.960383 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:29.959573 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-podinfo\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:30.046749 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:30.046715 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xfrx\" (UniqueName: \"kubernetes.io/projected/efe5fedb-d8ee-418b-89d6-1d1891e436df-kube-api-access-4xfrx\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:30.047646 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:30.047625 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-token\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:30.059593 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:30.059564 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:30.076379 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:30.076353 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4f477c-5c9jv" Oct 30 18:53:30.158398 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:30.157842 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd8446ae-527e-451f-81f6-88f7afd3f457-kube-api-access-fhkbd\" (UniqueName: \"kubernetes.io/projected/bd8446ae-527e-451f-81f6-88f7afd3f457-kube-api-access-fhkbd\") pod \"bd8446ae-527e-451f-81f6-88f7afd3f457\" (UID: \"bd8446ae-527e-451f-81f6-88f7afd3f457\") " Oct 30 18:53:30.158398 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:30.157945 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd8446ae-527e-451f-81f6-88f7afd3f457-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd8446ae-527e-451f-81f6-88f7afd3f457-maas-api-openshift-ca\") pod \"bd8446ae-527e-451f-81f6-88f7afd3f457\" (UID: \"bd8446ae-527e-451f-81f6-88f7afd3f457\") " Oct 30 18:53:30.158398 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:30.158001 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd8446ae-527e-451f-81f6-88f7afd3f457-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd8446ae-527e-451f-81f6-88f7afd3f457-maas-api-xks-ca\") pod \"bd8446ae-527e-451f-81f6-88f7afd3f457\" (UID: \"bd8446ae-527e-451f-81f6-88f7afd3f457\") " Oct 30 18:53:30.158746 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:30.158491 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd8446ae-527e-451f-81f6-88f7afd3f457-maas-api-openshift-ca" pod "bd8446ae-527e-451f-81f6-88f7afd3f457" (UID: "bd8446ae-527e-451f-81f6-88f7afd3f457"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:30.158746 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:30.158613 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd8446ae-527e-451f-81f6-88f7afd3f457-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:30.161576 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:30.161540 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8446ae-527e-451f-81f6-88f7afd3f457-kube-api-access-fhkbd" pod "bd8446ae-527e-451f-81f6-88f7afd3f457" (UID: "bd8446ae-527e-451f-81f6-88f7afd3f457"). InnerVolumeSpecName "kube-api-access-fhkbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:30.163962 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:30.163924 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd8446ae-527e-451f-81f6-88f7afd3f457-maas-api-xks-ca" pod "bd8446ae-527e-451f-81f6-88f7afd3f457" (UID: "bd8446ae-527e-451f-81f6-88f7afd3f457"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:30.260083 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:30.260048 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd8446ae-527e-451f-81f6-88f7afd3f457-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:30.260083 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:30.260075 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhkbd\" (UniqueName: \"kubernetes.io/projected/bd8446ae-527e-451f-81f6-88f7afd3f457-kube-api-access-fhkbd\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:30.352934 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:30.352874 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" podStartSLOduration=2.352855583 podStartE2EDuration="2.352855583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:30.133500463 +0000 UTC m=+1170.160605429" watchObservedRunningTime="2026-10-30 18:53:30.352855583 +0000 UTC m=+1170.379960542" Oct 30 18:53:30.544404 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:53:30.544370 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe5fedb_d8ee_418b_89d6_1d1891e436df.slice/crio-c1f5cff59ecd53a6acc9a69a25ff52903d6cf38769b18ea4b3e225ddcc0c3c5d WatchSource:0}: Error finding container c1f5cff59ecd53a6acc9a69a25ff52903d6cf38769b18ea4b3e225ddcc0c3c5d: Status 404 returned error can't find the container with id c1f5cff59ecd53a6acc9a69a25ff52903d6cf38769b18ea4b3e225ddcc0c3c5d Oct 30 18:53:30.546203 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:30.546179 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2"] Oct 30 18:53:30.547432 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:30.547397 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:53:30.547502 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:30.547479 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:53:30.547538 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:30.547522 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:53:30.604688 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:30.604651 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:30.606557 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:30.606500 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:53:30.606557 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:30.606546 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" podUID="56bbae65-edbd-48ad-b1f6-7c5da8d2d53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:53:30.886627 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:30.886536 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" event={"ID":"efe5fedb-d8ee-418b-89d6-1d1891e436df","Type":"ContainerStarted","Data":"92c63cdca82c8b7236eb30ed333fc21a000e9d30d3b8a08a87ea165d0ae45046"} Oct 30 18:53:30.886627 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:30.886600 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" event={"ID":"efe5fedb-d8ee-418b-89d6-1d1891e436df","Type":"ContainerStarted","Data":"c1f5cff59ecd53a6acc9a69a25ff52903d6cf38769b18ea4b3e225ddcc0c3c5d"} Oct 30 18:53:30.888099 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:30.888078 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4f477c-5c9jv" Oct 30 18:53:30.888246 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:30.888115 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf4f477c-5c9jv" event={"ID":"bd8446ae-527e-451f-81f6-88f7afd3f457","Type":"ContainerDied","Data":"7f2e54f692c766a6e7347925d619fd46f94b6d2720b51e941bedd5ccf613bf92"} Oct 30 18:53:30.888246 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:30.888183 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f2e54f692c766a6e7347925d619fd46f94b6d2720b51e941bedd5ccf613bf92" Oct 30 18:53:31.040171 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:31.039672 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" podStartSLOduration=2.039645797 podStartE2EDuration="2.039645797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:31.009746134 +0000 UTC m=+1171.036851093" watchObservedRunningTime="2026-10-30 18:53:31.039645797 +0000 UTC m=+1171.066750757" Oct 30 18:53:31.060802 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:31.060775 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:31.062664 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:31.062625 4531 patch_prober.go:29] interesting pod/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:53:31.062798 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:31.062681 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" podUID="efe5fedb-d8ee-418b-89d6-1d1891e436df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 18:53:31.090712 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:31.090678 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b161d939-k5xqz" Oct 30 18:53:31.173239 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:31.170824 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae1ac14-8378-4501-ba25-4e59f1e674ac-kube-api-access-clw96\" (UniqueName: \"kubernetes.io/projected/5ae1ac14-8378-4501-ba25-4e59f1e674ac-kube-api-access-clw96\") pod \"5ae1ac14-8378-4501-ba25-4e59f1e674ac\" (UID: \"5ae1ac14-8378-4501-ba25-4e59f1e674ac\") " Oct 30 18:53:31.173239 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:31.170887 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ae1ac14-8378-4501-ba25-4e59f1e674ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae1ac14-8378-4501-ba25-4e59f1e674ac-maas-api-xks-ca\") pod \"5ae1ac14-8378-4501-ba25-4e59f1e674ac\" (UID: \"5ae1ac14-8378-4501-ba25-4e59f1e674ac\") " Oct 30 18:53:31.173239 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:31.171118 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ae1ac14-8378-4501-ba25-4e59f1e674ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae1ac14-8378-4501-ba25-4e59f1e674ac-maas-api-openshift-ca\") pod \"5ae1ac14-8378-4501-ba25-4e59f1e674ac\" (UID: \"5ae1ac14-8378-4501-ba25-4e59f1e674ac\") " Oct 30 18:53:31.175835 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:31.175287 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae1ac14-8378-4501-ba25-4e59f1e674ac-maas-api-openshift-ca" pod "5ae1ac14-8378-4501-ba25-4e59f1e674ac" (UID: "5ae1ac14-8378-4501-ba25-4e59f1e674ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:31.175835 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:31.175755 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ae1ac14-8378-4501-ba25-4e59f1e674ac-maas-api-xks-ca" pod "5ae1ac14-8378-4501-ba25-4e59f1e674ac" (UID: "5ae1ac14-8378-4501-ba25-4e59f1e674ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:31.176902 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:31.176874 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae1ac14-8378-4501-ba25-4e59f1e674ac-kube-api-access-clw96" pod "5ae1ac14-8378-4501-ba25-4e59f1e674ac" (UID: "5ae1ac14-8378-4501-ba25-4e59f1e674ac"). InnerVolumeSpecName "kube-api-access-clw96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:31.273893 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:31.273809 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae1ac14-8378-4501-ba25-4e59f1e674ac-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:31.273893 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:31.273849 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clw96\" (UniqueName: \"kubernetes.io/projected/5ae1ac14-8378-4501-ba25-4e59f1e674ac-kube-api-access-clw96\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:31.273893 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:31.273866 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae1ac14-8378-4501-ba25-4e59f1e674ac-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:31.608219 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:31.604348 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:53:31.608219 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:31.604406 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" podUID="56bbae65-edbd-48ad-b1f6-7c5da8d2d53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:53:31.895611 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:31.895516 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b161d939-k5xqz" Oct 30 18:53:31.898260 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:31.898225 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b161d939-k5xqz" event={"ID":"5ae1ac14-8378-4501-ba25-4e59f1e674ac","Type":"ContainerDied","Data":"88d2a5f3b37fb39c50430095c4106335433fc21e23dc97afeca06981ac34e05b"} Oct 30 18:53:31.898508 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:31.898492 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88d2a5f3b37fb39c50430095c4106335433fc21e23dc97afeca06981ac34e05b" Oct 30 18:53:32.063157 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:32.062644 4531 patch_prober.go:29] interesting pod/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:53:32.063157 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:32.062709 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" podUID="efe5fedb-d8ee-418b-89d6-1d1891e436df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 18:53:32.604281 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:32.604239 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:53:32.604451 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:32.604295 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" podUID="56bbae65-edbd-48ad-b1f6-7c5da8d2d53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:53:33.064869 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:33.064322 4531 patch_prober.go:29] interesting pod/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:53:33.064869 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:33.064395 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" podUID="efe5fedb-d8ee-418b-89d6-1d1891e436df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 18:53:33.604938 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:33.604886 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:53:33.605101 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:33.604948 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" podUID="56bbae65-edbd-48ad-b1f6-7c5da8d2d53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:53:34.061647 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:34.061606 4531 patch_prober.go:29] interesting pod/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:53:34.061941 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:34.061913 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" podUID="efe5fedb-d8ee-418b-89d6-1d1891e436df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 18:53:34.604521 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:34.604490 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:53:34.604900 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:34.604542 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" podUID="56bbae65-edbd-48ad-b1f6-7c5da8d2d53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:53:35.064648 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:35.064605 4531 patch_prober.go:29] interesting pod/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:53:35.064916 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:35.064884 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" podUID="efe5fedb-d8ee-418b-89d6-1d1891e436df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 18:53:35.604212 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:35.604175 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:53:35.604377 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:35.604236 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" podUID="56bbae65-edbd-48ad-b1f6-7c5da8d2d53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:53:36.060813 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:36.060777 4531 patch_prober.go:29] interesting pod/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:53:36.061221 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:36.060831 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" podUID="efe5fedb-d8ee-418b-89d6-1d1891e436df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 18:53:36.677113 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:36.677081 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:36.677506 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:36.677484 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:36.678257 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:36.678236 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 18:53:37.060813 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:37.060780 4531 patch_prober.go:29] interesting pod/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:53:37.061107 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:37.060841 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" podUID="efe5fedb-d8ee-418b-89d6-1d1891e436df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 18:53:38.112222 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:38.112185 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:38.112886 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:38.112864 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:38.114283 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:38.114257 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 18:53:43.398417 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.398369 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh"] Oct 30 18:53:43.399010 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.398987 4531 cpu_manager.go:414] "containerMap: removing container" podUID="bd8446ae-527e-451f-81f6-88f7afd3f457" containerName="revoke-keys" Oct 30 18:53:43.399097 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.399014 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd8446ae-527e-451f-81f6-88f7afd3f457" containerName="revoke-keys" Oct 30 18:53:43.399097 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.399029 4531 cpu_manager.go:414] "containerMap: removing container" podUID="5ae1ac14-8378-4501-ba25-4e59f1e674ac" containerName="revoke-keys" Oct 30 18:53:43.399097 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.399039 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ae1ac14-8378-4501-ba25-4e59f1e674ac" containerName="revoke-keys" Oct 30 18:53:43.399283 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.399212 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd8446ae-527e-451f-81f6-88f7afd3f457" containerName="revoke-keys" Oct 30 18:53:43.399283 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.399225 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ae1ac14-8378-4501-ba25-4e59f1e674ac" containerName="revoke-keys" Oct 30 18:53:43.403712 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.403687 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.464931 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.464900 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c985eea-b9c8k"] Oct 30 18:53:43.473464 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.469697 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c985eea-b9c8k" Oct 30 18:53:43.509801 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.509766 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh"] Oct 30 18:53:43.525898 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.525855 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-workload-certs\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.526099 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.525912 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-envoy\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.526099 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.526015 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-data\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.526099 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.526046 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv5p6\" (UniqueName: \"kubernetes.io/projected/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-kube-api-access-cv5p6\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.526099 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.526075 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-credential-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.526312 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.526100 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-podinfo\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.526312 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.526180 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-token\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.526380 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.526342 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-workload-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.526380 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.526375 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istiod-ca-cert\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.551566 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.551533 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:53:43.551754 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.551653 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:53:43.551754 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.551737 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:53:43.554853 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.554805 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:53:43.559518 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.559489 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c985eea-b9c8k"] Oct 30 18:53:43.627457 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.627420 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-data\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.627672 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.627469 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vm4v\" (UniqueName: \"kubernetes.io/projected/670e2e17-779c-4357-b360-4eb70afb8131-kube-api-access-9vm4v\") pod \"maas-api-revoke-sub-9c985eea-b9c8k\" (UID: \"670e2e17-779c-4357-b360-4eb70afb8131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c985eea-b9c8k" Oct 30 18:53:43.627672 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.627495 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv5p6\" (UniqueName: \"kubernetes.io/projected/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-kube-api-access-cv5p6\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.627672 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.627523 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-credential-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.627672 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.627601 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-podinfo\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.627672 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.627632 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-token\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.627672 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.627670 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670e2e17-779c-4357-b360-4eb70afb8131-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c985eea-b9c8k\" (UID: \"670e2e17-779c-4357-b360-4eb70afb8131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c985eea-b9c8k" Oct 30 18:53:43.628019 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.627744 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-workload-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.628019 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.627780 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istiod-ca-cert\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.628019 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.627871 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-workload-certs\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.628019 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.627912 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670e2e17-779c-4357-b360-4eb70afb8131-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c985eea-b9c8k\" (UID: \"670e2e17-779c-4357-b360-4eb70afb8131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c985eea-b9c8k" Oct 30 18:53:43.628019 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.627944 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-data\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.628019 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.627976 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-envoy\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.628309 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.628067 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-credential-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.628309 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.628266 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-workload-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.628309 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.628269 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-workload-certs\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.628730 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.628637 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istiod-ca-cert\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.630418 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.630396 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-podinfo\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.630509 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.630433 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-envoy\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.689238 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.689123 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-token\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.689395 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.689361 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv5p6\" (UniqueName: \"kubernetes.io/projected/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-kube-api-access-cv5p6\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.717116 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.717074 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:43.729096 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.729067 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vm4v\" (UniqueName: \"kubernetes.io/projected/670e2e17-779c-4357-b360-4eb70afb8131-kube-api-access-9vm4v\") pod \"maas-api-revoke-sub-9c985eea-b9c8k\" (UID: \"670e2e17-779c-4357-b360-4eb70afb8131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c985eea-b9c8k" Oct 30 18:53:43.729204 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.729177 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670e2e17-779c-4357-b360-4eb70afb8131-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c985eea-b9c8k\" (UID: \"670e2e17-779c-4357-b360-4eb70afb8131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c985eea-b9c8k" Oct 30 18:53:43.729310 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.729295 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670e2e17-779c-4357-b360-4eb70afb8131-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c985eea-b9c8k\" (UID: \"670e2e17-779c-4357-b360-4eb70afb8131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c985eea-b9c8k" Oct 30 18:53:43.729810 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.729775 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670e2e17-779c-4357-b360-4eb70afb8131-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c985eea-b9c8k\" (UID: \"670e2e17-779c-4357-b360-4eb70afb8131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c985eea-b9c8k" Oct 30 18:53:43.731735 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.731712 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670e2e17-779c-4357-b360-4eb70afb8131-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c985eea-b9c8k\" (UID: \"670e2e17-779c-4357-b360-4eb70afb8131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c985eea-b9c8k" Oct 30 18:53:43.794612 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:43.794579 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vm4v\" (UniqueName: \"kubernetes.io/projected/670e2e17-779c-4357-b360-4eb70afb8131-kube-api-access-9vm4v\") pod \"maas-api-revoke-sub-9c985eea-b9c8k\" (UID: \"670e2e17-779c-4357-b360-4eb70afb8131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c985eea-b9c8k" Oct 30 18:53:44.087039 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:44.087001 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c985eea-b9c8k" Oct 30 18:53:44.154578 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:44.150064 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh"] Oct 30 18:53:44.155084 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:53:44.154877 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8192f408_5c6f_4c8e_b1b2_2729c7e4e8e6.slice/crio-0c0077a1d8c5283fb0bf221491efc15b7e0ba0879a9008cf88f2703357e09dd5 WatchSource:0}: Error finding container 0c0077a1d8c5283fb0bf221491efc15b7e0ba0879a9008cf88f2703357e09dd5: Status 404 returned error can't find the container with id 0c0077a1d8c5283fb0bf221491efc15b7e0ba0879a9008cf88f2703357e09dd5 Oct 30 18:53:44.160048 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:44.159736 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:53:44.160048 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:44.159810 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:53:44.160048 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:44.159839 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:53:44.467102 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:44.467069 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c985eea-b9c8k"] Oct 30 18:53:44.469422 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:53:44.469392 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod670e2e17_779c_4357_b360_4eb70afb8131.slice/crio-3c715d70c0b259ce71592aca85ddbe0e67b8c52113b4929c8d75ed3273d85da5 WatchSource:0}: Error finding container 3c715d70c0b259ce71592aca85ddbe0e67b8c52113b4929c8d75ed3273d85da5: Status 404 returned error can't find the container with id 3c715d70c0b259ce71592aca85ddbe0e67b8c52113b4929c8d75ed3273d85da5 Oct 30 18:53:44.646811 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:44.646774 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7964c56e-5rsbb"] Oct 30 18:53:44.651970 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:44.651948 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7964c56e-5rsbb" Oct 30 18:53:44.709639 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:44.709602 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7964c56e-5rsbb"] Oct 30 18:53:44.744193 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:44.740096 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7964c56e-5rsbb\" (UID: \"8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7964c56e-5rsbb" Oct 30 18:53:44.744193 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:44.740370 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7964c56e-5rsbb\" (UID: \"8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7964c56e-5rsbb" Oct 30 18:53:44.744193 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:44.740512 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk48x\" (UniqueName: \"kubernetes.io/projected/8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1-kube-api-access-hk48x\") pod \"maas-api-revoke-sub-7964c56e-5rsbb\" (UID: \"8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7964c56e-5rsbb" Oct 30 18:53:44.846195 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:44.846035 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7964c56e-5rsbb\" (UID: \"8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7964c56e-5rsbb" Oct 30 18:53:44.846400 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:44.846257 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk48x\" (UniqueName: \"kubernetes.io/projected/8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1-kube-api-access-hk48x\") pod \"maas-api-revoke-sub-7964c56e-5rsbb\" (UID: \"8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7964c56e-5rsbb" Oct 30 18:53:44.846701 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:44.846680 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7964c56e-5rsbb\" (UID: \"8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7964c56e-5rsbb" Oct 30 18:53:44.847767 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:44.847716 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7964c56e-5rsbb\" (UID: \"8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7964c56e-5rsbb" Oct 30 18:53:44.848111 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:44.848061 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7964c56e-5rsbb\" (UID: \"8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7964c56e-5rsbb" Oct 30 18:53:44.907295 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:44.907260 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk48x\" (UniqueName: \"kubernetes.io/projected/8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1-kube-api-access-hk48x\") pod \"maas-api-revoke-sub-7964c56e-5rsbb\" (UID: \"8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7964c56e-5rsbb" Oct 30 18:53:44.966432 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:44.966397 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7964c56e-5rsbb" Oct 30 18:53:44.990194 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:44.990114 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c985eea-b9c8k" event={"ID":"670e2e17-779c-4357-b360-4eb70afb8131","Type":"ContainerStarted","Data":"346fd32a340c7d0daaa61f1e4727755847efede003a4bf519d6669c889babf0f"} Oct 30 18:53:44.990377 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:44.990227 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c985eea-b9c8k" event={"ID":"670e2e17-779c-4357-b360-4eb70afb8131","Type":"ContainerStarted","Data":"3c715d70c0b259ce71592aca85ddbe0e67b8c52113b4929c8d75ed3273d85da5"} Oct 30 18:53:44.992026 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:44.991995 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" event={"ID":"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6","Type":"ContainerStarted","Data":"bcd8a7d479e675dbf063838c670abb5163e72cd4035c52154867fede86c59f4d"} Oct 30 18:53:44.992200 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:44.992046 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" event={"ID":"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6","Type":"ContainerStarted","Data":"0c0077a1d8c5283fb0bf221491efc15b7e0ba0879a9008cf88f2703357e09dd5"} Oct 30 18:53:45.109794 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:45.109730 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c985eea-b9c8k" podStartSLOduration=2.109712651 podStartE2EDuration="2.109712651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:45.108795326 +0000 UTC m=+1185.135900285" watchObservedRunningTime="2026-10-30 18:53:45.109712651 +0000 UTC m=+1185.136817611" Oct 30 18:53:45.243041 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:45.242964 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" podStartSLOduration=2.242942563 podStartE2EDuration="2.242942563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:45.227545252 +0000 UTC m=+1185.254650212" watchObservedRunningTime="2026-10-30 18:53:45.242942563 +0000 UTC m=+1185.270047523" Oct 30 18:53:45.366847 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:45.366817 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7964c56e-5rsbb"] Oct 30 18:53:45.367794 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:53:45.367768 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8daf35c6_a29b_4bbd_a2ea_0fb37ad163e1.slice/crio-3558c2a0bce57f7eec711bc004b7b57c0715fa6e9854074af4eaebe9099fdbc1 WatchSource:0}: Error finding container 3558c2a0bce57f7eec711bc004b7b57c0715fa6e9854074af4eaebe9099fdbc1: Status 404 returned error can't find the container with id 3558c2a0bce57f7eec711bc004b7b57c0715fa6e9854074af4eaebe9099fdbc1 Oct 30 18:53:45.727093 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:45.727061 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:45.727810 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:45.727787 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:53:45.727982 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:45.727958 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" podUID="8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 18:53:46.004878 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:46.004836 4531 generic.go:356] "Generic (PLEG): container finished" podID="670e2e17-779c-4357-b360-4eb70afb8131" containerID="346fd32a340c7d0daaa61f1e4727755847efede003a4bf519d6669c889babf0f" exitCode=0 Oct 30 18:53:46.005163 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:46.005093 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c985eea-b9c8k" event={"ID":"670e2e17-779c-4357-b360-4eb70afb8131","Type":"ContainerDied","Data":"346fd32a340c7d0daaa61f1e4727755847efede003a4bf519d6669c889babf0f"} Oct 30 18:53:46.007566 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:46.007541 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7964c56e-5rsbb" event={"ID":"8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1","Type":"ContainerStarted","Data":"ce5534153763aa9d9842acdfc824b5ccfb80af20b4d96ec8021e216d3e7ae740"} Oct 30 18:53:46.007763 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:46.007746 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7964c56e-5rsbb" event={"ID":"8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1","Type":"ContainerStarted","Data":"3558c2a0bce57f7eec711bc004b7b57c0715fa6e9854074af4eaebe9099fdbc1"} Oct 30 18:53:46.280214 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:46.279015 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7964c56e-5rsbb" podStartSLOduration=2.278995198 podStartE2EDuration="2.278995198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:46.269447086 +0000 UTC m=+1186.296552045" watchObservedRunningTime="2026-10-30 18:53:46.278995198 +0000 UTC m=+1186.306100158" Oct 30 18:53:46.729156 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:46.719602 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:53:46.729156 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:46.719669 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" podUID="8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 18:53:47.061532 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:47.061492 4531 generic.go:356] "Generic (PLEG): container finished" podID="8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1" containerID="ce5534153763aa9d9842acdfc824b5ccfb80af20b4d96ec8021e216d3e7ae740" exitCode=0 Oct 30 18:53:47.063102 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:47.063071 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7964c56e-5rsbb" event={"ID":"8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1","Type":"ContainerDied","Data":"ce5534153763aa9d9842acdfc824b5ccfb80af20b4d96ec8021e216d3e7ae740"} Oct 30 18:53:47.827152 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:47.826911 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:53:47.827152 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:47.826971 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" podUID="8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 18:53:47.972789 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:47.955274 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" podUID="5a868137-eb41-433c-ade6-58eb8c96d623" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:48.036191 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:48.036092 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c985eea-b9c8k" Oct 30 18:53:48.121882 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:48.120471 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c985eea-b9c8k" Oct 30 18:53:48.140781 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:48.126745 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c985eea-b9c8k" event={"ID":"670e2e17-779c-4357-b360-4eb70afb8131","Type":"ContainerDied","Data":"3c715d70c0b259ce71592aca85ddbe0e67b8c52113b4929c8d75ed3273d85da5"} Oct 30 18:53:48.140781 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:48.126832 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c715d70c0b259ce71592aca85ddbe0e67b8c52113b4929c8d75ed3273d85da5" Oct 30 18:53:48.280684 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:48.280642 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/670e2e17-779c-4357-b360-4eb70afb8131-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670e2e17-779c-4357-b360-4eb70afb8131-maas-api-openshift-ca\") pod \"670e2e17-779c-4357-b360-4eb70afb8131\" (UID: \"670e2e17-779c-4357-b360-4eb70afb8131\") " Oct 30 18:53:48.281622 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:48.281380 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/670e2e17-779c-4357-b360-4eb70afb8131-maas-api-openshift-ca" pod "670e2e17-779c-4357-b360-4eb70afb8131" (UID: "670e2e17-779c-4357-b360-4eb70afb8131"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:48.294424 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:48.281857 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/670e2e17-779c-4357-b360-4eb70afb8131-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670e2e17-779c-4357-b360-4eb70afb8131-maas-api-xks-ca\") pod \"670e2e17-779c-4357-b360-4eb70afb8131\" (UID: \"670e2e17-779c-4357-b360-4eb70afb8131\") " Oct 30 18:53:48.307590 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:48.294897 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/670e2e17-779c-4357-b360-4eb70afb8131-kube-api-access-9vm4v\" (UniqueName: \"kubernetes.io/projected/670e2e17-779c-4357-b360-4eb70afb8131-kube-api-access-9vm4v\") pod \"670e2e17-779c-4357-b360-4eb70afb8131\" (UID: \"670e2e17-779c-4357-b360-4eb70afb8131\") " Oct 30 18:53:48.343865 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:48.340698 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/670e2e17-779c-4357-b360-4eb70afb8131-kube-api-access-9vm4v" pod "670e2e17-779c-4357-b360-4eb70afb8131" (UID: "670e2e17-779c-4357-b360-4eb70afb8131"). InnerVolumeSpecName "kube-api-access-9vm4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:48.357159 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:48.351347 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670e2e17-779c-4357-b360-4eb70afb8131-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:48.380158 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:48.366274 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/670e2e17-779c-4357-b360-4eb70afb8131-maas-api-xks-ca" pod "670e2e17-779c-4357-b360-4eb70afb8131" (UID: "670e2e17-779c-4357-b360-4eb70afb8131"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:48.453154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:48.452966 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670e2e17-779c-4357-b360-4eb70afb8131-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:48.453154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:48.453009 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vm4v\" (UniqueName: \"kubernetes.io/projected/670e2e17-779c-4357-b360-4eb70afb8131-kube-api-access-9vm4v\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:48.684113 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:48.684084 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7964c56e-5rsbb" Oct 30 18:53:48.736177 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:48.720618 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:53:48.736177 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:48.720720 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" podUID="8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 18:53:48.759770 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:48.759731 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1-kube-api-access-hk48x\" (UniqueName: \"kubernetes.io/projected/8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1-kube-api-access-hk48x\") pod \"8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1\" (UID: \"8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1\") " Oct 30 18:53:48.759770 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:48.759776 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1-maas-api-xks-ca\") pod \"8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1\" (UID: \"8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1\") " Oct 30 18:53:48.760041 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:48.759987 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1-maas-api-openshift-ca\") pod \"8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1\" (UID: \"8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1\") " Oct 30 18:53:48.760377 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:48.760344 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1-maas-api-openshift-ca" pod "8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1" (UID: "8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:48.760695 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:48.760555 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:48.762143 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:48.762104 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1-maas-api-xks-ca" pod "8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1" (UID: "8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:48.762455 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:48.762438 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1-kube-api-access-hk48x" pod "8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1" (UID: "8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1"). InnerVolumeSpecName "kube-api-access-hk48x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:48.862075 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:48.861980 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk48x\" (UniqueName: \"kubernetes.io/projected/8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1-kube-api-access-hk48x\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:48.862075 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:48.862024 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:49.144952 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:49.144852 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7964c56e-5rsbb" Oct 30 18:53:49.144952 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:49.144889 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7964c56e-5rsbb" event={"ID":"8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1","Type":"ContainerDied","Data":"3558c2a0bce57f7eec711bc004b7b57c0715fa6e9854074af4eaebe9099fdbc1"} Oct 30 18:53:49.144952 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:49.144942 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3558c2a0bce57f7eec711bc004b7b57c0715fa6e9854074af4eaebe9099fdbc1" Oct 30 18:53:49.729158 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:49.719582 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:53:49.729158 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:49.719644 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" podUID="8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 18:53:50.718596 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:50.718551 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:53:50.719113 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:50.718608 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" podUID="8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 18:53:52.759368 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:52.759318 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:53:52.759899 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:52.759404 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" podUID="8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:53.058919 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:53.058821 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:53.059296 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:53.059268 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:53.060222 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:53.060178 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 18:53:54.268377 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:54.268343 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce18950a-dps9s"] Oct 30 18:53:54.270072 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:54.268783 4531 cpu_manager.go:414] "containerMap: removing container" podUID="670e2e17-779c-4357-b360-4eb70afb8131" containerName="revoke-keys" Oct 30 18:53:54.270072 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:54.268794 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="670e2e17-779c-4357-b360-4eb70afb8131" containerName="revoke-keys" Oct 30 18:53:54.270072 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:54.268805 4531 cpu_manager.go:414] "containerMap: removing container" podUID="8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1" containerName="revoke-keys" Oct 30 18:53:54.270072 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:54.268810 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1" containerName="revoke-keys" Oct 30 18:53:54.270072 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:54.268878 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="670e2e17-779c-4357-b360-4eb70afb8131" containerName="revoke-keys" Oct 30 18:53:54.270072 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:54.268885 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1" containerName="revoke-keys" Oct 30 18:53:54.272017 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:54.271999 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce18950a-dps9s" Oct 30 18:53:54.329503 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:54.329404 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63efe720-284f-4533-9f8d-ebfd08dc07d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce18950a-dps9s\" (UID: \"63efe720-284f-4533-9f8d-ebfd08dc07d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce18950a-dps9s" Oct 30 18:53:54.329503 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:54.329472 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzhn7\" (UniqueName: \"kubernetes.io/projected/63efe720-284f-4533-9f8d-ebfd08dc07d1-kube-api-access-tzhn7\") pod \"maas-api-revoke-sub-ce18950a-dps9s\" (UID: \"63efe720-284f-4533-9f8d-ebfd08dc07d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce18950a-dps9s" Oct 30 18:53:54.329699 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:54.329589 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63efe720-284f-4533-9f8d-ebfd08dc07d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce18950a-dps9s\" (UID: \"63efe720-284f-4533-9f8d-ebfd08dc07d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce18950a-dps9s" Oct 30 18:53:54.375899 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:54.375866 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:53:54.375899 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:54.375866 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:53:54.375899 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:54.375881 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:53:54.376197 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:54.376029 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:53:54.379069 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:54.379048 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce18950a-dps9s"] Oct 30 18:53:54.430325 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:54.430292 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63efe720-284f-4533-9f8d-ebfd08dc07d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce18950a-dps9s\" (UID: \"63efe720-284f-4533-9f8d-ebfd08dc07d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce18950a-dps9s" Oct 30 18:53:54.430521 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:54.430341 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzhn7\" (UniqueName: \"kubernetes.io/projected/63efe720-284f-4533-9f8d-ebfd08dc07d1-kube-api-access-tzhn7\") pod \"maas-api-revoke-sub-ce18950a-dps9s\" (UID: \"63efe720-284f-4533-9f8d-ebfd08dc07d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce18950a-dps9s" Oct 30 18:53:54.430521 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:54.430376 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63efe720-284f-4533-9f8d-ebfd08dc07d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce18950a-dps9s\" (UID: \"63efe720-284f-4533-9f8d-ebfd08dc07d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce18950a-dps9s" Oct 30 18:53:54.431063 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:54.431039 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63efe720-284f-4533-9f8d-ebfd08dc07d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce18950a-dps9s\" (UID: \"63efe720-284f-4533-9f8d-ebfd08dc07d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce18950a-dps9s" Oct 30 18:53:54.432913 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:54.432895 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63efe720-284f-4533-9f8d-ebfd08dc07d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce18950a-dps9s\" (UID: \"63efe720-284f-4533-9f8d-ebfd08dc07d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce18950a-dps9s" Oct 30 18:53:54.492084 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:54.492056 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzhn7\" (UniqueName: \"kubernetes.io/projected/63efe720-284f-4533-9f8d-ebfd08dc07d1-kube-api-access-tzhn7\") pod \"maas-api-revoke-sub-ce18950a-dps9s\" (UID: \"63efe720-284f-4533-9f8d-ebfd08dc07d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce18950a-dps9s" Oct 30 18:53:54.585617 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:54.585531 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce18950a-dps9s" Oct 30 18:53:54.967584 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:54.967556 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce18950a-dps9s"] Oct 30 18:53:54.969273 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:53:54.969246 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63efe720_284f_4533_9f8d_ebfd08dc07d1.slice/crio-bd44c2ec9f7fde72cc3640d205063f050aa7724d35cdffb37dcaf9f454adb84e WatchSource:0}: Error finding container bd44c2ec9f7fde72cc3640d205063f050aa7724d35cdffb37dcaf9f454adb84e: Status 404 returned error can't find the container with id bd44c2ec9f7fde72cc3640d205063f050aa7724d35cdffb37dcaf9f454adb84e Oct 30 18:53:55.183228 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:55.183192 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce18950a-dps9s" event={"ID":"63efe720-284f-4533-9f8d-ebfd08dc07d1","Type":"ContainerStarted","Data":"bd44c2ec9f7fde72cc3640d205063f050aa7724d35cdffb37dcaf9f454adb84e"} Oct 30 18:53:55.429275 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:55.429229 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb6fa702-28sj4"] Oct 30 18:53:55.435231 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:55.435197 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6fa702-28sj4" Oct 30 18:53:55.490344 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:55.490270 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb6fa702-28sj4"] Oct 30 18:53:55.542888 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:55.542829 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e964bfe-cdfd-4baf-8d32-15c35847afd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb6fa702-28sj4\" (UID: \"8e964bfe-cdfd-4baf-8d32-15c35847afd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6fa702-28sj4" Oct 30 18:53:55.543070 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:55.542961 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e964bfe-cdfd-4baf-8d32-15c35847afd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb6fa702-28sj4\" (UID: \"8e964bfe-cdfd-4baf-8d32-15c35847afd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6fa702-28sj4" Oct 30 18:53:55.543070 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:55.543052 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk4rb\" (UniqueName: \"kubernetes.io/projected/8e964bfe-cdfd-4baf-8d32-15c35847afd6-kube-api-access-zk4rb\") pod \"maas-api-revoke-sub-cb6fa702-28sj4\" (UID: \"8e964bfe-cdfd-4baf-8d32-15c35847afd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6fa702-28sj4" Oct 30 18:53:55.643724 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:55.643692 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e964bfe-cdfd-4baf-8d32-15c35847afd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb6fa702-28sj4\" (UID: \"8e964bfe-cdfd-4baf-8d32-15c35847afd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6fa702-28sj4" Oct 30 18:53:55.643939 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:55.643780 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk4rb\" (UniqueName: \"kubernetes.io/projected/8e964bfe-cdfd-4baf-8d32-15c35847afd6-kube-api-access-zk4rb\") pod \"maas-api-revoke-sub-cb6fa702-28sj4\" (UID: \"8e964bfe-cdfd-4baf-8d32-15c35847afd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6fa702-28sj4" Oct 30 18:53:55.643939 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:55.643835 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e964bfe-cdfd-4baf-8d32-15c35847afd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb6fa702-28sj4\" (UID: \"8e964bfe-cdfd-4baf-8d32-15c35847afd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6fa702-28sj4" Oct 30 18:53:55.644516 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:55.644487 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e964bfe-cdfd-4baf-8d32-15c35847afd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb6fa702-28sj4\" (UID: \"8e964bfe-cdfd-4baf-8d32-15c35847afd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6fa702-28sj4" Oct 30 18:53:55.646316 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:55.646299 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e964bfe-cdfd-4baf-8d32-15c35847afd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb6fa702-28sj4\" (UID: \"8e964bfe-cdfd-4baf-8d32-15c35847afd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6fa702-28sj4" Oct 30 18:53:55.705846 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:55.705813 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk4rb\" (UniqueName: \"kubernetes.io/projected/8e964bfe-cdfd-4baf-8d32-15c35847afd6-kube-api-access-zk4rb\") pod \"maas-api-revoke-sub-cb6fa702-28sj4\" (UID: \"8e964bfe-cdfd-4baf-8d32-15c35847afd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6fa702-28sj4" Oct 30 18:53:55.754980 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:55.754933 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6fa702-28sj4" Oct 30 18:53:56.148706 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:56.148676 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb6fa702-28sj4"] Oct 30 18:53:56.150049 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:53:56.150017 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e964bfe_cdfd_4baf_8d32_15c35847afd6.slice/crio-3e7110ce970edcb0fb060784ff551ca127176a05da6fb709426e25fb5660200c WatchSource:0}: Error finding container 3e7110ce970edcb0fb060784ff551ca127176a05da6fb709426e25fb5660200c: Status 404 returned error can't find the container with id 3e7110ce970edcb0fb060784ff551ca127176a05da6fb709426e25fb5660200c Oct 30 18:53:56.191052 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:56.191010 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6fa702-28sj4" event={"ID":"8e964bfe-cdfd-4baf-8d32-15c35847afd6","Type":"ContainerStarted","Data":"3e7110ce970edcb0fb060784ff551ca127176a05da6fb709426e25fb5660200c"} Oct 30 18:53:56.192602 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:56.192576 4531 generic.go:356] "Generic (PLEG): container finished" podID="63efe720-284f-4533-9f8d-ebfd08dc07d1" containerID="92568be9d5f827ab6d7f3ea8a3fbb67bd074c3aa955399ae3e03136783ebc566" exitCode=0 Oct 30 18:53:56.192739 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:56.192661 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce18950a-dps9s" event={"ID":"63efe720-284f-4533-9f8d-ebfd08dc07d1","Type":"ContainerDied","Data":"92568be9d5f827ab6d7f3ea8a3fbb67bd074c3aa955399ae3e03136783ebc566"} Oct 30 18:53:56.397726 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:53:56.397636 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:53:57.200302 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:57.200270 4531 generic.go:356] "Generic (PLEG): container finished" podID="8e964bfe-cdfd-4baf-8d32-15c35847afd6" containerID="2535c12555bc33445094450bda76fe71c3c85368d755d4db0fbd8a34b3d00bca" exitCode=0 Oct 30 18:53:57.200720 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:57.200350 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6fa702-28sj4" event={"ID":"8e964bfe-cdfd-4baf-8d32-15c35847afd6","Type":"ContainerDied","Data":"2535c12555bc33445094450bda76fe71c3c85368d755d4db0fbd8a34b3d00bca"} Oct 30 18:53:57.350805 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:57.350774 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce18950a-dps9s" Oct 30 18:53:57.465077 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:57.465042 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63efe720-284f-4533-9f8d-ebfd08dc07d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63efe720-284f-4533-9f8d-ebfd08dc07d1-maas-api-xks-ca\") pod \"63efe720-284f-4533-9f8d-ebfd08dc07d1\" (UID: \"63efe720-284f-4533-9f8d-ebfd08dc07d1\") " Oct 30 18:53:57.465330 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:57.465221 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63efe720-284f-4533-9f8d-ebfd08dc07d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63efe720-284f-4533-9f8d-ebfd08dc07d1-maas-api-openshift-ca\") pod \"63efe720-284f-4533-9f8d-ebfd08dc07d1\" (UID: \"63efe720-284f-4533-9f8d-ebfd08dc07d1\") " Oct 30 18:53:57.465853 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:57.465696 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63efe720-284f-4533-9f8d-ebfd08dc07d1-maas-api-openshift-ca" pod "63efe720-284f-4533-9f8d-ebfd08dc07d1" (UID: "63efe720-284f-4533-9f8d-ebfd08dc07d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:57.465943 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:57.465828 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63efe720-284f-4533-9f8d-ebfd08dc07d1-kube-api-access-tzhn7\" (UniqueName: \"kubernetes.io/projected/63efe720-284f-4533-9f8d-ebfd08dc07d1-kube-api-access-tzhn7\") pod \"63efe720-284f-4533-9f8d-ebfd08dc07d1\" (UID: \"63efe720-284f-4533-9f8d-ebfd08dc07d1\") " Oct 30 18:53:57.466382 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:57.466341 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63efe720-284f-4533-9f8d-ebfd08dc07d1-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:57.468177 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:57.468118 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63efe720-284f-4533-9f8d-ebfd08dc07d1-kube-api-access-tzhn7" pod "63efe720-284f-4533-9f8d-ebfd08dc07d1" (UID: "63efe720-284f-4533-9f8d-ebfd08dc07d1"). InnerVolumeSpecName "kube-api-access-tzhn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:57.468496 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:57.468456 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63efe720-284f-4533-9f8d-ebfd08dc07d1-maas-api-xks-ca" pod "63efe720-284f-4533-9f8d-ebfd08dc07d1" (UID: "63efe720-284f-4533-9f8d-ebfd08dc07d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:57.567807 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:57.567772 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63efe720-284f-4533-9f8d-ebfd08dc07d1-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:57.567807 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:57.567803 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzhn7\" (UniqueName: \"kubernetes.io/projected/63efe720-284f-4533-9f8d-ebfd08dc07d1-kube-api-access-tzhn7\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:58.207089 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:58.207057 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce18950a-dps9s" Oct 30 18:53:58.207502 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:58.207056 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce18950a-dps9s" event={"ID":"63efe720-284f-4533-9f8d-ebfd08dc07d1","Type":"ContainerDied","Data":"bd44c2ec9f7fde72cc3640d205063f050aa7724d35cdffb37dcaf9f454adb84e"} Oct 30 18:53:58.207502 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:58.207201 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd44c2ec9f7fde72cc3640d205063f050aa7724d35cdffb37dcaf9f454adb84e" Oct 30 18:53:58.356196 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:58.356121 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6fa702-28sj4" Oct 30 18:53:58.477328 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:58.477297 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e964bfe-cdfd-4baf-8d32-15c35847afd6-kube-api-access-zk4rb\" (UniqueName: \"kubernetes.io/projected/8e964bfe-cdfd-4baf-8d32-15c35847afd6-kube-api-access-zk4rb\") pod \"8e964bfe-cdfd-4baf-8d32-15c35847afd6\" (UID: \"8e964bfe-cdfd-4baf-8d32-15c35847afd6\") " Oct 30 18:53:58.477520 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:58.477404 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e964bfe-cdfd-4baf-8d32-15c35847afd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e964bfe-cdfd-4baf-8d32-15c35847afd6-maas-api-xks-ca\") pod \"8e964bfe-cdfd-4baf-8d32-15c35847afd6\" (UID: \"8e964bfe-cdfd-4baf-8d32-15c35847afd6\") " Oct 30 18:53:58.477520 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:58.477441 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e964bfe-cdfd-4baf-8d32-15c35847afd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e964bfe-cdfd-4baf-8d32-15c35847afd6-maas-api-openshift-ca\") pod \"8e964bfe-cdfd-4baf-8d32-15c35847afd6\" (UID: \"8e964bfe-cdfd-4baf-8d32-15c35847afd6\") " Oct 30 18:53:58.477832 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:58.477807 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e964bfe-cdfd-4baf-8d32-15c35847afd6-maas-api-openshift-ca" pod "8e964bfe-cdfd-4baf-8d32-15c35847afd6" (UID: "8e964bfe-cdfd-4baf-8d32-15c35847afd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:58.479633 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:58.479598 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e964bfe-cdfd-4baf-8d32-15c35847afd6-maas-api-xks-ca" pod "8e964bfe-cdfd-4baf-8d32-15c35847afd6" (UID: "8e964bfe-cdfd-4baf-8d32-15c35847afd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:58.479935 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:58.479915 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e964bfe-cdfd-4baf-8d32-15c35847afd6-kube-api-access-zk4rb" pod "8e964bfe-cdfd-4baf-8d32-15c35847afd6" (UID: "8e964bfe-cdfd-4baf-8d32-15c35847afd6"). InnerVolumeSpecName "kube-api-access-zk4rb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:58.579239 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:58.579202 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e964bfe-cdfd-4baf-8d32-15c35847afd6-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:58.579239 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:58.579231 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e964bfe-cdfd-4baf-8d32-15c35847afd6-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:58.579239 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:58.579244 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk4rb\" (UniqueName: \"kubernetes.io/projected/8e964bfe-cdfd-4baf-8d32-15c35847afd6-kube-api-access-zk4rb\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:53:59.214906 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:59.214879 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6fa702-28sj4" Oct 30 18:53:59.215415 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:59.214914 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6fa702-28sj4" event={"ID":"8e964bfe-cdfd-4baf-8d32-15c35847afd6","Type":"ContainerDied","Data":"3e7110ce970edcb0fb060784ff551ca127176a05da6fb709426e25fb5660200c"} Oct 30 18:53:59.215415 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:53:59.214966 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e7110ce970edcb0fb060784ff551ca127176a05da6fb709426e25fb5660200c" Oct 30 18:54:11.929347 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:11.929261 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a50c88dc-cwmgl"] Oct 30 18:54:11.929817 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:11.929728 4531 cpu_manager.go:414] "containerMap: removing container" podUID="63efe720-284f-4533-9f8d-ebfd08dc07d1" containerName="revoke-keys" Oct 30 18:54:11.929817 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:11.929744 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63efe720-284f-4533-9f8d-ebfd08dc07d1" containerName="revoke-keys" Oct 30 18:54:11.929817 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:11.929764 4531 cpu_manager.go:414] "containerMap: removing container" podUID="8e964bfe-cdfd-4baf-8d32-15c35847afd6" containerName="revoke-keys" Oct 30 18:54:11.929817 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:11.929770 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e964bfe-cdfd-4baf-8d32-15c35847afd6" containerName="revoke-keys" Oct 30 18:54:11.929953 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:11.929862 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="63efe720-284f-4533-9f8d-ebfd08dc07d1" containerName="revoke-keys" Oct 30 18:54:11.929953 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:11.929871 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e964bfe-cdfd-4baf-8d32-15c35847afd6" containerName="revoke-keys" Oct 30 18:54:11.934823 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:11.934800 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50c88dc-cwmgl" Oct 30 18:54:11.981965 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:11.981923 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a50c88dc-cwmgl\" (UID: \"d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50c88dc-cwmgl" Oct 30 18:54:11.982185 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:11.982061 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a50c88dc-cwmgl\" (UID: \"d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50c88dc-cwmgl" Oct 30 18:54:11.982185 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:11.982111 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcg59\" (UniqueName: \"kubernetes.io/projected/d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef-kube-api-access-pcg59\") pod \"maas-api-revoke-sub-a50c88dc-cwmgl\" (UID: \"d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50c88dc-cwmgl" Oct 30 18:54:11.996080 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:11.996050 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a50c88dc-cwmgl"] Oct 30 18:54:12.014044 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.014013 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:54:12.014044 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.014013 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:54:12.014270 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.014075 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:54:12.040517 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.040487 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:54:12.083019 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.082979 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a50c88dc-cwmgl\" (UID: \"d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50c88dc-cwmgl" Oct 30 18:54:12.083252 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.083029 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcg59\" (UniqueName: \"kubernetes.io/projected/d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef-kube-api-access-pcg59\") pod \"maas-api-revoke-sub-a50c88dc-cwmgl\" (UID: \"d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50c88dc-cwmgl" Oct 30 18:54:12.083252 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.083166 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a50c88dc-cwmgl\" (UID: \"d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50c88dc-cwmgl" Oct 30 18:54:12.083745 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.083717 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a50c88dc-cwmgl\" (UID: \"d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50c88dc-cwmgl" Oct 30 18:54:12.085727 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.085704 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a50c88dc-cwmgl\" (UID: \"d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50c88dc-cwmgl" Oct 30 18:54:12.115026 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.114995 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4844a61-9fzvb"] Oct 30 18:54:12.118005 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.117977 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4844a61-9fzvb"] Oct 30 18:54:12.145576 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.145546 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcg59\" (UniqueName: \"kubernetes.io/projected/d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef-kube-api-access-pcg59\") pod \"maas-api-revoke-sub-a50c88dc-cwmgl\" (UID: \"d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50c88dc-cwmgl" Oct 30 18:54:12.249046 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.249022 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50c88dc-cwmgl" Oct 30 18:54:12.596478 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.596404 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f73bd23a-hngd6"] Oct 30 18:54:12.601219 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.601196 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73bd23a-hngd6" Oct 30 18:54:12.625076 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.625052 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a50c88dc-cwmgl"] Oct 30 18:54:12.626500 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:54:12.626475 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c7cd2f_7cce_421b_8a6d_f2577b7a7bef.slice/crio-bac133b743784a98ea71a31b70ba9431c70c866516e73ed50160aacea4ea9165 WatchSource:0}: Error finding container bac133b743784a98ea71a31b70ba9431c70c866516e73ed50160aacea4ea9165: Status 404 returned error can't find the container with id bac133b743784a98ea71a31b70ba9431c70c866516e73ed50160aacea4ea9165 Oct 30 18:54:12.661465 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.661435 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f73bd23a-hngd6"] Oct 30 18:54:12.689928 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.689891 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89bwn\" (UniqueName: \"kubernetes.io/projected/3e86dc9c-db4d-478a-8e7c-2b04900e5c04-kube-api-access-89bwn\") pod \"maas-api-revoke-sub-f73bd23a-hngd6\" (UID: \"3e86dc9c-db4d-478a-8e7c-2b04900e5c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73bd23a-hngd6" Oct 30 18:54:12.690153 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.689948 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e86dc9c-db4d-478a-8e7c-2b04900e5c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f73bd23a-hngd6\" (UID: \"3e86dc9c-db4d-478a-8e7c-2b04900e5c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73bd23a-hngd6" Oct 30 18:54:12.690153 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.690034 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e86dc9c-db4d-478a-8e7c-2b04900e5c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f73bd23a-hngd6\" (UID: \"3e86dc9c-db4d-478a-8e7c-2b04900e5c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73bd23a-hngd6" Oct 30 18:54:12.791087 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.791042 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89bwn\" (UniqueName: \"kubernetes.io/projected/3e86dc9c-db4d-478a-8e7c-2b04900e5c04-kube-api-access-89bwn\") pod \"maas-api-revoke-sub-f73bd23a-hngd6\" (UID: \"3e86dc9c-db4d-478a-8e7c-2b04900e5c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73bd23a-hngd6" Oct 30 18:54:12.791320 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.791109 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e86dc9c-db4d-478a-8e7c-2b04900e5c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f73bd23a-hngd6\" (UID: \"3e86dc9c-db4d-478a-8e7c-2b04900e5c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73bd23a-hngd6" Oct 30 18:54:12.791320 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.791190 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e86dc9c-db4d-478a-8e7c-2b04900e5c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f73bd23a-hngd6\" (UID: \"3e86dc9c-db4d-478a-8e7c-2b04900e5c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73bd23a-hngd6" Oct 30 18:54:12.791750 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.791728 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e86dc9c-db4d-478a-8e7c-2b04900e5c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f73bd23a-hngd6\" (UID: \"3e86dc9c-db4d-478a-8e7c-2b04900e5c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73bd23a-hngd6" Oct 30 18:54:12.793724 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.793701 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e86dc9c-db4d-478a-8e7c-2b04900e5c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f73bd23a-hngd6\" (UID: \"3e86dc9c-db4d-478a-8e7c-2b04900e5c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73bd23a-hngd6" Oct 30 18:54:12.853506 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.853425 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89bwn\" (UniqueName: \"kubernetes.io/projected/3e86dc9c-db4d-478a-8e7c-2b04900e5c04-kube-api-access-89bwn\") pod \"maas-api-revoke-sub-f73bd23a-hngd6\" (UID: \"3e86dc9c-db4d-478a-8e7c-2b04900e5c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73bd23a-hngd6" Oct 30 18:54:12.922140 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:12.922092 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73bd23a-hngd6" Oct 30 18:54:13.298719 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:13.298685 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f73bd23a-hngd6"] Oct 30 18:54:13.300338 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:54:13.300314 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e86dc9c_db4d_478a_8e7c_2b04900e5c04.slice/crio-8619ee2c7039dbe01aec09d364e70b0f2823b060187e2cb109afa2ec509f3f72 WatchSource:0}: Error finding container 8619ee2c7039dbe01aec09d364e70b0f2823b060187e2cb109afa2ec509f3f72: Status 404 returned error can't find the container with id 8619ee2c7039dbe01aec09d364e70b0f2823b060187e2cb109afa2ec509f3f72 Oct 30 18:54:13.373104 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:13.373069 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73bd23a-hngd6" event={"ID":"3e86dc9c-db4d-478a-8e7c-2b04900e5c04","Type":"ContainerStarted","Data":"8619ee2c7039dbe01aec09d364e70b0f2823b060187e2cb109afa2ec509f3f72"} Oct 30 18:54:13.375290 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:13.375263 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50c88dc-cwmgl" event={"ID":"d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef","Type":"ContainerStarted","Data":"1ed82a1b3e43af1fb84f31c07aea0d8ba3fd6db635326e0c554e000c1c4eb7e8"} Oct 30 18:54:13.375464 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:13.375318 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50c88dc-cwmgl" event={"ID":"d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef","Type":"ContainerStarted","Data":"bac133b743784a98ea71a31b70ba9431c70c866516e73ed50160aacea4ea9165"} Oct 30 18:54:13.505821 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:13.505766 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50c88dc-cwmgl" podStartSLOduration=2.505753018 podStartE2EDuration="2.505753018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:13.504401321 +0000 UTC m=+1213.531506282" watchObservedRunningTime="2026-10-30 18:54:13.505753018 +0000 UTC m=+1213.532857976" Oct 30 18:54:14.098493 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:14.098456 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c05a1547-0c46-4b53-b122-b9d7ae9588c4" path="/var/lib/kubelet/pods/c05a1547-0c46-4b53-b122-b9d7ae9588c4/volumes" Oct 30 18:54:14.382984 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:14.382897 4531 generic.go:356] "Generic (PLEG): container finished" podID="3e86dc9c-db4d-478a-8e7c-2b04900e5c04" containerID="7e3c6512fc690ee217a6df05cd20929d235d3ba712f5ffc1777b0ab700cfb06e" exitCode=0 Oct 30 18:54:14.383399 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:14.382980 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73bd23a-hngd6" event={"ID":"3e86dc9c-db4d-478a-8e7c-2b04900e5c04","Type":"ContainerDied","Data":"7e3c6512fc690ee217a6df05cd20929d235d3ba712f5ffc1777b0ab700cfb06e"} Oct 30 18:54:14.384719 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:14.384693 4531 generic.go:356] "Generic (PLEG): container finished" podID="d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef" containerID="1ed82a1b3e43af1fb84f31c07aea0d8ba3fd6db635326e0c554e000c1c4eb7e8" exitCode=0 Oct 30 18:54:14.384889 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:14.384780 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50c88dc-cwmgl" event={"ID":"d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef","Type":"ContainerDied","Data":"1ed82a1b3e43af1fb84f31c07aea0d8ba3fd6db635326e0c554e000c1c4eb7e8"} Oct 30 18:54:15.070759 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.070716 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q"] Oct 30 18:54:15.074611 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.074587 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.118346 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.118312 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-token\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.118551 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.118362 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-data\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.118551 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.118453 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-workload-certs\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.118551 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.118489 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfs7g\" (UniqueName: \"kubernetes.io/projected/4adfad0a-5914-442f-afe1-c7f9c2f70849-kube-api-access-pfs7g\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.118706 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.118594 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-envoy\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.118706 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.118622 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-credential-socket\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.118814 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.118749 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4adfad0a-5914-442f-afe1-c7f9c2f70849-istiod-ca-cert\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.118865 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.118795 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-workload-socket\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.118865 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.118844 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-podinfo\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.220177 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.220122 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4adfad0a-5914-442f-afe1-c7f9c2f70849-istiod-ca-cert\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.220387 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.220210 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-workload-socket\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.220387 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.220227 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-podinfo\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.220387 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.220294 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-token\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.220387 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.220360 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-data\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.220620 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.220414 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-workload-certs\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.220620 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.220442 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfs7g\" (UniqueName: \"kubernetes.io/projected/4adfad0a-5914-442f-afe1-c7f9c2f70849-kube-api-access-pfs7g\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.220620 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.220500 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-envoy\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.220620 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.220528 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-credential-socket\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.220826 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.220644 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-workload-socket\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.221341 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.220991 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4adfad0a-5914-442f-afe1-c7f9c2f70849-istiod-ca-cert\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.221341 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.221254 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-workload-certs\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.221341 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.221289 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-data\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.222199 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.222171 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-credential-socket\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.223715 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.223694 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-envoy\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.224232 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.224041 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-podinfo\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.279916 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.279880 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q"] Oct 30 18:54:15.283019 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.282905 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-token\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.283019 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.282933 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfs7g\" (UniqueName: \"kubernetes.io/projected/4adfad0a-5914-442f-afe1-c7f9c2f70849-kube-api-access-pfs7g\") pod \"e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.389780 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.389698 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:15.570551 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.570524 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73bd23a-hngd6" Oct 30 18:54:15.596834 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.596804 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50c88dc-cwmgl" Oct 30 18:54:15.625054 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.625015 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e86dc9c-db4d-478a-8e7c-2b04900e5c04-kube-api-access-89bwn\" (UniqueName: \"kubernetes.io/projected/3e86dc9c-db4d-478a-8e7c-2b04900e5c04-kube-api-access-89bwn\") pod \"3e86dc9c-db4d-478a-8e7c-2b04900e5c04\" (UID: \"3e86dc9c-db4d-478a-8e7c-2b04900e5c04\") " Oct 30 18:54:15.625257 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.625067 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e86dc9c-db4d-478a-8e7c-2b04900e5c04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e86dc9c-db4d-478a-8e7c-2b04900e5c04-maas-api-xks-ca\") pod \"3e86dc9c-db4d-478a-8e7c-2b04900e5c04\" (UID: \"3e86dc9c-db4d-478a-8e7c-2b04900e5c04\") " Oct 30 18:54:15.625257 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.625201 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e86dc9c-db4d-478a-8e7c-2b04900e5c04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e86dc9c-db4d-478a-8e7c-2b04900e5c04-maas-api-openshift-ca\") pod \"3e86dc9c-db4d-478a-8e7c-2b04900e5c04\" (UID: \"3e86dc9c-db4d-478a-8e7c-2b04900e5c04\") " Oct 30 18:54:15.625802 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.625773 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e86dc9c-db4d-478a-8e7c-2b04900e5c04-maas-api-openshift-ca" pod "3e86dc9c-db4d-478a-8e7c-2b04900e5c04" (UID: "3e86dc9c-db4d-478a-8e7c-2b04900e5c04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:15.627604 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.627571 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e86dc9c-db4d-478a-8e7c-2b04900e5c04-maas-api-xks-ca" pod "3e86dc9c-db4d-478a-8e7c-2b04900e5c04" (UID: "3e86dc9c-db4d-478a-8e7c-2b04900e5c04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:54:15.627733 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.627713 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e86dc9c-db4d-478a-8e7c-2b04900e5c04-kube-api-access-89bwn" pod "3e86dc9c-db4d-478a-8e7c-2b04900e5c04" (UID: "3e86dc9c-db4d-478a-8e7c-2b04900e5c04"). InnerVolumeSpecName "kube-api-access-89bwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:15.725978 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.725892 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef-maas-api-xks-ca\") pod \"d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef\" (UID: \"d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef\") " Oct 30 18:54:15.726171 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.725981 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef-kube-api-access-pcg59\" (UniqueName: \"kubernetes.io/projected/d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef-kube-api-access-pcg59\") pod \"d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef\" (UID: \"d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef\") " Oct 30 18:54:15.726171 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.726089 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef-maas-api-openshift-ca\") pod \"d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef\" (UID: \"d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef\") " Oct 30 18:54:15.726517 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.726479 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef-maas-api-openshift-ca" pod "d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef" (UID: "d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:15.726861 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.726839 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e86dc9c-db4d-478a-8e7c-2b04900e5c04-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:54:15.726951 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.726865 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:54:15.726951 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.726880 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89bwn\" (UniqueName: \"kubernetes.io/projected/3e86dc9c-db4d-478a-8e7c-2b04900e5c04-kube-api-access-89bwn\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:54:15.726951 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.726895 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e86dc9c-db4d-478a-8e7c-2b04900e5c04-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:54:15.728323 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.728302 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef-kube-api-access-pcg59" pod "d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef" (UID: "d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef"). InnerVolumeSpecName "kube-api-access-pcg59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:15.728323 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.728306 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef-maas-api-xks-ca" pod "d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef" (UID: "d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:54:15.839642 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.839602 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:54:15.839642 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.839638 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcg59\" (UniqueName: \"kubernetes.io/projected/d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef-kube-api-access-pcg59\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:54:15.855059 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.855028 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q"] Oct 30 18:54:15.855342 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:54:15.855319 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4adfad0a_5914_442f_afe1_c7f9c2f70849.slice/crio-15b425da6db2f5b6632d79cc5810df5225906f3782eb1ec86f13cc9d5ab067cb WatchSource:0}: Error finding container 15b425da6db2f5b6632d79cc5810df5225906f3782eb1ec86f13cc9d5ab067cb: Status 404 returned error can't find the container with id 15b425da6db2f5b6632d79cc5810df5225906f3782eb1ec86f13cc9d5ab067cb Oct 30 18:54:15.857779 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.857743 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:54:15.857867 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.857813 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:54:15.857867 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:15.857839 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:54:16.399499 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:16.399429 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73bd23a-hngd6" Oct 30 18:54:16.399499 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:16.399462 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73bd23a-hngd6" event={"ID":"3e86dc9c-db4d-478a-8e7c-2b04900e5c04","Type":"ContainerDied","Data":"8619ee2c7039dbe01aec09d364e70b0f2823b060187e2cb109afa2ec509f3f72"} Oct 30 18:54:16.399978 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:16.399520 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8619ee2c7039dbe01aec09d364e70b0f2823b060187e2cb109afa2ec509f3f72" Oct 30 18:54:16.400835 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:16.400809 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50c88dc-cwmgl" event={"ID":"d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef","Type":"ContainerDied","Data":"bac133b743784a98ea71a31b70ba9431c70c866516e73ed50160aacea4ea9165"} Oct 30 18:54:16.400975 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:16.400858 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bac133b743784a98ea71a31b70ba9431c70c866516e73ed50160aacea4ea9165" Oct 30 18:54:16.400975 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:16.400823 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a50c88dc-cwmgl" Oct 30 18:54:16.402632 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:16.402606 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" event={"ID":"4adfad0a-5914-442f-afe1-c7f9c2f70849","Type":"ContainerStarted","Data":"8b5a37b986be2c06b49f1d324cea81721b09431bc76379c7d7d3cee807f7eb19"} Oct 30 18:54:16.402748 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:16.402650 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" event={"ID":"4adfad0a-5914-442f-afe1-c7f9c2f70849","Type":"ContainerStarted","Data":"15b425da6db2f5b6632d79cc5810df5225906f3782eb1ec86f13cc9d5ab067cb"} Oct 30 18:54:16.571776 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:16.571715 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" podStartSLOduration=1.571700361 podStartE2EDuration="1.571700361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:16.52640764 +0000 UTC m=+1216.553512636" watchObservedRunningTime="2026-10-30 18:54:16.571700361 +0000 UTC m=+1216.598805320" Oct 30 18:54:17.390212 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:17.390180 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:17.391723 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:17.391697 4531 patch_prober.go:29] interesting pod/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:54:17.391856 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:17.391749 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" podUID="4adfad0a-5914-442f-afe1-c7f9c2f70849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 18:54:18.390945 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:18.390903 4531 patch_prober.go:29] interesting pod/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:54:18.391465 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:18.390966 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" podUID="4adfad0a-5914-442f-afe1-c7f9c2f70849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 18:54:19.391090 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.391055 4531 patch_prober.go:29] interesting pod/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:54:19.391497 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.391106 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" podUID="4adfad0a-5914-442f-afe1-c7f9c2f70849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 18:54:19.443720 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.443681 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx"] Oct 30 18:54:19.444157 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.444117 4531 cpu_manager.go:414] "containerMap: removing container" podUID="d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef" containerName="revoke-keys" Oct 30 18:54:19.444293 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.444193 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef" containerName="revoke-keys" Oct 30 18:54:19.444293 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.444226 4531 cpu_manager.go:414] "containerMap: removing container" podUID="3e86dc9c-db4d-478a-8e7c-2b04900e5c04" containerName="revoke-keys" Oct 30 18:54:19.444293 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.444236 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e86dc9c-db4d-478a-8e7c-2b04900e5c04" containerName="revoke-keys" Oct 30 18:54:19.444391 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.444377 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef" containerName="revoke-keys" Oct 30 18:54:19.444424 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.444391 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e86dc9c-db4d-478a-8e7c-2b04900e5c04" containerName="revoke-keys" Oct 30 18:54:19.448004 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.447986 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" Oct 30 18:54:19.509583 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.509554 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2ade09\"/\"distinct-w0-2ade09-kserve-self-signed-certs\"" Oct 30 18:54:19.554928 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.554897 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx"] Oct 30 18:54:19.582088 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.582054 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-dshm\") pod \"distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx\" (UID: \"471f1171-e71f-4da2-862d-5d232c45d6cc\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" Oct 30 18:54:19.582088 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.582098 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-model-cache\") pod \"distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx\" (UID: \"471f1171-e71f-4da2-862d-5d232c45d6cc\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" Oct 30 18:54:19.582343 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.582318 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-home\") pod \"distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx\" (UID: \"471f1171-e71f-4da2-862d-5d232c45d6cc\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" Oct 30 18:54:19.582564 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.582540 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25mxq\" (UniqueName: \"kubernetes.io/projected/471f1171-e71f-4da2-862d-5d232c45d6cc-kube-api-access-25mxq\") pod \"distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx\" (UID: \"471f1171-e71f-4da2-862d-5d232c45d6cc\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" Oct 30 18:54:19.582705 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.582606 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-tmp-dir\") pod \"distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx\" (UID: \"471f1171-e71f-4da2-862d-5d232c45d6cc\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" Oct 30 18:54:19.582768 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.582692 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/471f1171-e71f-4da2-862d-5d232c45d6cc-tls-certs\") pod \"distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx\" (UID: \"471f1171-e71f-4da2-862d-5d232c45d6cc\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" Oct 30 18:54:19.684058 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.683976 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-dshm\") pod \"distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx\" (UID: \"471f1171-e71f-4da2-862d-5d232c45d6cc\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" Oct 30 18:54:19.684058 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.684010 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-model-cache\") pod \"distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx\" (UID: \"471f1171-e71f-4da2-862d-5d232c45d6cc\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" Oct 30 18:54:19.684330 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.684091 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-home\") pod \"distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx\" (UID: \"471f1171-e71f-4da2-862d-5d232c45d6cc\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" Oct 30 18:54:19.684330 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.684207 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25mxq\" (UniqueName: \"kubernetes.io/projected/471f1171-e71f-4da2-862d-5d232c45d6cc-kube-api-access-25mxq\") pod \"distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx\" (UID: \"471f1171-e71f-4da2-862d-5d232c45d6cc\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" Oct 30 18:54:19.684330 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.684254 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-tmp-dir\") pod \"distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx\" (UID: \"471f1171-e71f-4da2-862d-5d232c45d6cc\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" Oct 30 18:54:19.684330 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.684279 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/471f1171-e71f-4da2-862d-5d232c45d6cc-tls-certs\") pod \"distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx\" (UID: \"471f1171-e71f-4da2-862d-5d232c45d6cc\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" Oct 30 18:54:19.684563 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.684526 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-model-cache\") pod \"distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx\" (UID: \"471f1171-e71f-4da2-862d-5d232c45d6cc\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" Oct 30 18:54:19.684872 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.684834 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-home\") pod \"distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx\" (UID: \"471f1171-e71f-4da2-862d-5d232c45d6cc\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" Oct 30 18:54:19.684981 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.684899 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-tmp-dir\") pod \"distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx\" (UID: \"471f1171-e71f-4da2-862d-5d232c45d6cc\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" Oct 30 18:54:19.686894 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.686871 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-dshm\") pod \"distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx\" (UID: \"471f1171-e71f-4da2-862d-5d232c45d6cc\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" Oct 30 18:54:19.687072 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.686870 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/471f1171-e71f-4da2-862d-5d232c45d6cc-tls-certs\") pod \"distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx\" (UID: \"471f1171-e71f-4da2-862d-5d232c45d6cc\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" Oct 30 18:54:19.748150 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.747793 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25mxq\" (UniqueName: \"kubernetes.io/projected/471f1171-e71f-4da2-862d-5d232c45d6cc-kube-api-access-25mxq\") pod \"distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx\" (UID: \"471f1171-e71f-4da2-862d-5d232c45d6cc\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" Oct 30 18:54:19.760682 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:19.760375 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" Oct 30 18:54:20.167818 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:20.167599 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx"] Oct 30 18:54:20.390213 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:20.390179 4531 patch_prober.go:29] interesting pod/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:54:20.390390 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:20.390233 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" podUID="4adfad0a-5914-442f-afe1-c7f9c2f70849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 18:54:20.451542 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:20.451466 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" event={"ID":"471f1171-e71f-4da2-862d-5d232c45d6cc","Type":"ContainerStarted","Data":"0eacf94a6e4785a277442cd14bbb868e2264ddd5e4916191dfa5fff8b523c053"} Oct 30 18:54:20.451542 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:20.451505 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" event={"ID":"471f1171-e71f-4da2-862d-5d232c45d6cc","Type":"ContainerStarted","Data":"01f48d3942f7d3d50d133e67fc4e43909c498f937bfb0348e903d8791669b053"} Oct 30 18:54:20.570270 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:20.570210 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" podStartSLOduration=1.570196321 podStartE2EDuration="1.570196321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:20.567315815 +0000 UTC m=+1220.594420787" watchObservedRunningTime="2026-10-30 18:54:20.570196321 +0000 UTC m=+1220.597301280" Oct 30 18:54:20.761428 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:20.761401 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" Oct 30 18:54:21.390009 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:21.389980 4531 patch_prober.go:29] interesting pod/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:54:21.390215 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:21.390030 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" podUID="4adfad0a-5914-442f-afe1-c7f9c2f70849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 18:54:22.395155 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:22.394500 4531 patch_prober.go:29] interesting pod/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:54:22.395155 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:22.394562 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" podUID="4adfad0a-5914-442f-afe1-c7f9c2f70849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 18:54:22.983154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:22.976949 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1597-swvqx"] Oct 30 18:54:22.985870 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:22.985835 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1597-swvqx" Oct 30 18:54:23.057154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.053024 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:54:23.057154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.053097 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:54:23.057154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.055791 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb982b4-d179-4810-bbcd-e938c9797768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cbb1597-swvqx\" (UID: \"bfb982b4-d179-4810-bbcd-e938c9797768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1597-swvqx" Oct 30 18:54:23.057154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.055906 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb982b4-d179-4810-bbcd-e938c9797768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cbb1597-swvqx\" (UID: \"bfb982b4-d179-4810-bbcd-e938c9797768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1597-swvqx" Oct 30 18:54:23.057154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.055933 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpvh7\" (UniqueName: \"kubernetes.io/projected/bfb982b4-d179-4810-bbcd-e938c9797768-kube-api-access-dpvh7\") pod \"maas-api-revoke-sub-6cbb1597-swvqx\" (UID: \"bfb982b4-d179-4810-bbcd-e938c9797768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1597-swvqx" Oct 30 18:54:23.105155 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.081092 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1597-swvqx"] Oct 30 18:54:23.182151 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.173404 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:54:23.182151 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.173536 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:54:23.193176 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.184084 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb982b4-d179-4810-bbcd-e938c9797768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cbb1597-swvqx\" (UID: \"bfb982b4-d179-4810-bbcd-e938c9797768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1597-swvqx" Oct 30 18:54:23.193176 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.184157 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpvh7\" (UniqueName: \"kubernetes.io/projected/bfb982b4-d179-4810-bbcd-e938c9797768-kube-api-access-dpvh7\") pod \"maas-api-revoke-sub-6cbb1597-swvqx\" (UID: \"bfb982b4-d179-4810-bbcd-e938c9797768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1597-swvqx" Oct 30 18:54:23.193176 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.184325 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb982b4-d179-4810-bbcd-e938c9797768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cbb1597-swvqx\" (UID: \"bfb982b4-d179-4810-bbcd-e938c9797768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1597-swvqx" Oct 30 18:54:23.193176 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.185361 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb982b4-d179-4810-bbcd-e938c9797768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cbb1597-swvqx\" (UID: \"bfb982b4-d179-4810-bbcd-e938c9797768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1597-swvqx" Oct 30 18:54:23.212380 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.203190 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64384eda-fdfks"] Oct 30 18:54:23.369163 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.365220 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64384eda-fdfks"] Oct 30 18:54:23.489156 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.486825 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpvh7\" (UniqueName: \"kubernetes.io/projected/bfb982b4-d179-4810-bbcd-e938c9797768-kube-api-access-dpvh7\") pod \"maas-api-revoke-sub-6cbb1597-swvqx\" (UID: \"bfb982b4-d179-4810-bbcd-e938c9797768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1597-swvqx" Oct 30 18:54:23.508691 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.508657 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb982b4-d179-4810-bbcd-e938c9797768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cbb1597-swvqx\" (UID: \"bfb982b4-d179-4810-bbcd-e938c9797768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1597-swvqx" Oct 30 18:54:23.532182 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.508919 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" podUID="1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:54:23.628750 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.628644 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" podUID="586f370d-9916-43a9-ac7d-e77cc3fcb92f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:54:23.769205 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.769161 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" podUID="678454ff-4761-4bd4-b2a3-2b431bca7934" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:54:23.812925 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.812898 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1597-swvqx" Oct 30 18:54:23.852261 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.852229 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f947eab8-rrqtn"] Oct 30 18:54:23.855018 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.854980 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" podUID="1a5f18bf-96ba-4edf-a758-206a1bd926a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:54:23.857404 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.857388 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f947eab8-rrqtn"] Oct 30 18:54:23.857646 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.857633 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f947eab8-rrqtn" Oct 30 18:54:23.990927 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.990890 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57e07223-5536-45a0-931f-3c3874c63a67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f947eab8-rrqtn\" (UID: \"57e07223-5536-45a0-931f-3c3874c63a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f947eab8-rrqtn" Oct 30 18:54:23.991105 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.991090 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57e07223-5536-45a0-931f-3c3874c63a67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f947eab8-rrqtn\" (UID: \"57e07223-5536-45a0-931f-3c3874c63a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f947eab8-rrqtn" Oct 30 18:54:23.991501 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:23.991231 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t99z4\" (UniqueName: \"kubernetes.io/projected/57e07223-5536-45a0-931f-3c3874c63a67-kube-api-access-t99z4\") pod \"maas-api-revoke-sub-f947eab8-rrqtn\" (UID: \"57e07223-5536-45a0-931f-3c3874c63a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f947eab8-rrqtn" Oct 30 18:54:24.080670 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:24.080642 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e8bc64-47w74"] Oct 30 18:54:24.083370 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:24.083344 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e8bc64-47w74"] Oct 30 18:54:24.092987 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:24.092959 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57e07223-5536-45a0-931f-3c3874c63a67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f947eab8-rrqtn\" (UID: \"57e07223-5536-45a0-931f-3c3874c63a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f947eab8-rrqtn" Oct 30 18:54:24.093992 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:24.093970 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57e07223-5536-45a0-931f-3c3874c63a67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f947eab8-rrqtn\" (UID: \"57e07223-5536-45a0-931f-3c3874c63a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f947eab8-rrqtn" Oct 30 18:54:24.094577 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:24.094557 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t99z4\" (UniqueName: \"kubernetes.io/projected/57e07223-5536-45a0-931f-3c3874c63a67-kube-api-access-t99z4\") pod \"maas-api-revoke-sub-f947eab8-rrqtn\" (UID: \"57e07223-5536-45a0-931f-3c3874c63a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f947eab8-rrqtn" Oct 30 18:54:24.094812 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:24.093917 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57e07223-5536-45a0-931f-3c3874c63a67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f947eab8-rrqtn\" (UID: \"57e07223-5536-45a0-931f-3c3874c63a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f947eab8-rrqtn" Oct 30 18:54:24.109616 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:24.109587 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57e07223-5536-45a0-931f-3c3874c63a67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f947eab8-rrqtn\" (UID: \"57e07223-5536-45a0-931f-3c3874c63a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f947eab8-rrqtn" Oct 30 18:54:24.112507 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:24.112475 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc" path="/var/lib/kubelet/pods/4b9f9bfc-a3e1-4c34-9894-4ae0632b90cc/volumes" Oct 30 18:54:24.112988 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:24.112967 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95d71500-74ce-4e8a-a514-875df485349b" path="/var/lib/kubelet/pods/95d71500-74ce-4e8a-a514-875df485349b/volumes" Oct 30 18:54:24.157114 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:24.157029 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t99z4\" (UniqueName: \"kubernetes.io/projected/57e07223-5536-45a0-931f-3c3874c63a67-kube-api-access-t99z4\") pod \"maas-api-revoke-sub-f947eab8-rrqtn\" (UID: \"57e07223-5536-45a0-931f-3c3874c63a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f947eab8-rrqtn" Oct 30 18:54:24.191393 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:24.191358 4531 patch_prober.go:29] interesting pod/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:54:24.191568 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:24.191421 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" podUID="efe5fedb-d8ee-418b-89d6-1d1891e436df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:54:24.195186 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:24.195166 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f947eab8-rrqtn" Oct 30 18:54:24.311074 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:24.311050 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1597-swvqx"] Oct 30 18:54:24.311231 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:54:24.311207 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb982b4_d179_4810_bbcd_e938c9797768.slice/crio-f48cc25a2dceb720d0826708664516dfad77bd348fd5376d73f6c2434e0aa999 WatchSource:0}: Error finding container f48cc25a2dceb720d0826708664516dfad77bd348fd5376d73f6c2434e0aa999: Status 404 returned error can't find the container with id f48cc25a2dceb720d0826708664516dfad77bd348fd5376d73f6c2434e0aa999 Oct 30 18:54:24.404914 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:24.404879 4531 patch_prober.go:29] interesting pod/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:54:24.405062 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:24.404937 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" podUID="4adfad0a-5914-442f-afe1-c7f9c2f70849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:54:24.542577 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:24.542547 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1597-swvqx" event={"ID":"bfb982b4-d179-4810-bbcd-e938c9797768","Type":"ContainerStarted","Data":"f48cc25a2dceb720d0826708664516dfad77bd348fd5376d73f6c2434e0aa999"} Oct 30 18:54:24.580351 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:24.580317 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f947eab8-rrqtn"] Oct 30 18:54:24.581619 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:54:24.581582 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e07223_5536_45a0_931f_3c3874c63a67.slice/crio-bf4c91949ad3b5ba5dac92c81b3c3af8d8f257b69902e7611d3226bdb6a62ba1 WatchSource:0}: Error finding container bf4c91949ad3b5ba5dac92c81b3c3af8d8f257b69902e7611d3226bdb6a62ba1: Status 404 returned error can't find the container with id bf4c91949ad3b5ba5dac92c81b3c3af8d8f257b69902e7611d3226bdb6a62ba1 Oct 30 18:54:25.274976 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:25.274944 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:25.275593 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:25.275569 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:25.276366 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:25.276346 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 18:54:25.554838 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:25.554762 4531 generic.go:356] "Generic (PLEG): container finished" podID="bfb982b4-d179-4810-bbcd-e938c9797768" containerID="79fc3aec0fb617f9afeddf0a4836fed0764ecb508fe4c004fe547fd27b093a8b" exitCode=0 Oct 30 18:54:25.555238 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:25.554855 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1597-swvqx" event={"ID":"bfb982b4-d179-4810-bbcd-e938c9797768","Type":"ContainerDied","Data":"79fc3aec0fb617f9afeddf0a4836fed0764ecb508fe4c004fe547fd27b093a8b"} Oct 30 18:54:25.556572 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:25.556545 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f947eab8-rrqtn" event={"ID":"57e07223-5536-45a0-931f-3c3874c63a67","Type":"ContainerStarted","Data":"aaa59c807cea9f42da9de592e9c971f0a81a1d00297527569a066be2923770a6"} Oct 30 18:54:25.556699 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:25.556596 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f947eab8-rrqtn" event={"ID":"57e07223-5536-45a0-931f-3c3874c63a67","Type":"ContainerStarted","Data":"bf4c91949ad3b5ba5dac92c81b3c3af8d8f257b69902e7611d3226bdb6a62ba1"} Oct 30 18:54:25.787289 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:25.787241 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f947eab8-rrqtn" podStartSLOduration=2.787225818 podStartE2EDuration="2.787225818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:25.784327009 +0000 UTC m=+1225.811431969" watchObservedRunningTime="2026-10-30 18:54:25.787225818 +0000 UTC m=+1225.814330777" Oct 30 18:54:26.397256 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:54:26.397221 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:54:26.564009 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:26.563973 4531 generic.go:356] "Generic (PLEG): container finished" podID="57e07223-5536-45a0-931f-3c3874c63a67" containerID="aaa59c807cea9f42da9de592e9c971f0a81a1d00297527569a066be2923770a6" exitCode=0 Oct 30 18:54:26.564448 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:26.564064 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f947eab8-rrqtn" event={"ID":"57e07223-5536-45a0-931f-3c3874c63a67","Type":"ContainerDied","Data":"aaa59c807cea9f42da9de592e9c971f0a81a1d00297527569a066be2923770a6"} Oct 30 18:54:26.710742 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:26.710709 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1597-swvqx" Oct 30 18:54:26.733549 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:26.733525 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfb982b4-d179-4810-bbcd-e938c9797768-kube-api-access-dpvh7\" (UniqueName: \"kubernetes.io/projected/bfb982b4-d179-4810-bbcd-e938c9797768-kube-api-access-dpvh7\") pod \"bfb982b4-d179-4810-bbcd-e938c9797768\" (UID: \"bfb982b4-d179-4810-bbcd-e938c9797768\") " Oct 30 18:54:26.733692 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:26.733610 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfb982b4-d179-4810-bbcd-e938c9797768-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb982b4-d179-4810-bbcd-e938c9797768-maas-api-openshift-ca\") pod \"bfb982b4-d179-4810-bbcd-e938c9797768\" (UID: \"bfb982b4-d179-4810-bbcd-e938c9797768\") " Oct 30 18:54:26.733748 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:26.733695 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfb982b4-d179-4810-bbcd-e938c9797768-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb982b4-d179-4810-bbcd-e938c9797768-maas-api-xks-ca\") pod \"bfb982b4-d179-4810-bbcd-e938c9797768\" (UID: \"bfb982b4-d179-4810-bbcd-e938c9797768\") " Oct 30 18:54:26.734011 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:26.733985 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb982b4-d179-4810-bbcd-e938c9797768-maas-api-openshift-ca" pod "bfb982b4-d179-4810-bbcd-e938c9797768" (UID: "bfb982b4-d179-4810-bbcd-e938c9797768"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:26.734117 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:26.734094 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb982b4-d179-4810-bbcd-e938c9797768-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:54:26.735760 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:26.735740 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb982b4-d179-4810-bbcd-e938c9797768-kube-api-access-dpvh7" pod "bfb982b4-d179-4810-bbcd-e938c9797768" (UID: "bfb982b4-d179-4810-bbcd-e938c9797768"). InnerVolumeSpecName "kube-api-access-dpvh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:26.735931 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:26.735910 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb982b4-d179-4810-bbcd-e938c9797768-maas-api-xks-ca" pod "bfb982b4-d179-4810-bbcd-e938c9797768" (UID: "bfb982b4-d179-4810-bbcd-e938c9797768"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:54:26.835691 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:26.835656 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb982b4-d179-4810-bbcd-e938c9797768-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:54:26.835833 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:26.835698 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpvh7\" (UniqueName: \"kubernetes.io/projected/bfb982b4-d179-4810-bbcd-e938c9797768-kube-api-access-dpvh7\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:54:27.570919 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:27.570892 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1597-swvqx" Oct 30 18:54:27.571414 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:27.570923 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1597-swvqx" event={"ID":"bfb982b4-d179-4810-bbcd-e938c9797768","Type":"ContainerDied","Data":"f48cc25a2dceb720d0826708664516dfad77bd348fd5376d73f6c2434e0aa999"} Oct 30 18:54:27.571414 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:27.570970 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f48cc25a2dceb720d0826708664516dfad77bd348fd5376d73f6c2434e0aa999" Oct 30 18:54:27.719919 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:27.719899 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f947eab8-rrqtn" Oct 30 18:54:27.743939 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:27.743915 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57e07223-5536-45a0-931f-3c3874c63a67-kube-api-access-t99z4\" (UniqueName: \"kubernetes.io/projected/57e07223-5536-45a0-931f-3c3874c63a67-kube-api-access-t99z4\") pod \"57e07223-5536-45a0-931f-3c3874c63a67\" (UID: \"57e07223-5536-45a0-931f-3c3874c63a67\") " Oct 30 18:54:27.744113 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:27.744096 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57e07223-5536-45a0-931f-3c3874c63a67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57e07223-5536-45a0-931f-3c3874c63a67-maas-api-xks-ca\") pod \"57e07223-5536-45a0-931f-3c3874c63a67\" (UID: \"57e07223-5536-45a0-931f-3c3874c63a67\") " Oct 30 18:54:27.744206 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:27.744154 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57e07223-5536-45a0-931f-3c3874c63a67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57e07223-5536-45a0-931f-3c3874c63a67-maas-api-openshift-ca\") pod \"57e07223-5536-45a0-931f-3c3874c63a67\" (UID: \"57e07223-5536-45a0-931f-3c3874c63a67\") " Oct 30 18:54:27.744543 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:27.744512 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57e07223-5536-45a0-931f-3c3874c63a67-maas-api-openshift-ca" pod "57e07223-5536-45a0-931f-3c3874c63a67" (UID: "57e07223-5536-45a0-931f-3c3874c63a67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:27.744666 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:27.744595 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57e07223-5536-45a0-931f-3c3874c63a67-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:54:27.746084 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:27.746062 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e07223-5536-45a0-931f-3c3874c63a67-kube-api-access-t99z4" pod "57e07223-5536-45a0-931f-3c3874c63a67" (UID: "57e07223-5536-45a0-931f-3c3874c63a67"). InnerVolumeSpecName "kube-api-access-t99z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:27.746247 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:27.746231 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57e07223-5536-45a0-931f-3c3874c63a67-maas-api-xks-ca" pod "57e07223-5536-45a0-931f-3c3874c63a67" (UID: "57e07223-5536-45a0-931f-3c3874c63a67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:54:27.846165 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:27.846062 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57e07223-5536-45a0-931f-3c3874c63a67-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:54:27.846165 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:27.846094 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t99z4\" (UniqueName: \"kubernetes.io/projected/57e07223-5536-45a0-931f-3c3874c63a67-kube-api-access-t99z4\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:54:28.577850 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:28.577822 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f947eab8-rrqtn" Oct 30 18:54:28.578271 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:28.577816 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f947eab8-rrqtn" event={"ID":"57e07223-5536-45a0-931f-3c3874c63a67","Type":"ContainerDied","Data":"bf4c91949ad3b5ba5dac92c81b3c3af8d8f257b69902e7611d3226bdb6a62ba1"} Oct 30 18:54:28.578271 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:28.577952 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf4c91949ad3b5ba5dac92c81b3c3af8d8f257b69902e7611d3226bdb6a62ba1" Oct 30 18:54:29.761286 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:29.761250 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" Oct 30 18:54:29.774550 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:29.774527 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" Oct 30 18:54:30.051348 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:30.051272 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0900ec82-pb4jr"] Oct 30 18:54:30.056444 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:30.056421 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0900ec82-pb4jr"] Oct 30 18:54:30.099178 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:30.099151 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b7f47d2-b54d-4906-9597-5fe9188cd5c3" path="/var/lib/kubelet/pods/2b7f47d2-b54d-4906-9597-5fe9188cd5c3/volumes" Oct 30 18:54:30.600995 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:30.600968 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" Oct 30 18:54:39.052826 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.052796 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce96a591-4dcp6"] Oct 30 18:54:39.102182 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.102151 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce96a591-4dcp6"] Oct 30 18:54:39.268532 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.268503 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-185e5936-tvqhc"] Oct 30 18:54:39.269088 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.269073 4531 cpu_manager.go:414] "containerMap: removing container" podUID="bfb982b4-d179-4810-bbcd-e938c9797768" containerName="revoke-keys" Oct 30 18:54:39.269163 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.269090 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfb982b4-d179-4810-bbcd-e938c9797768" containerName="revoke-keys" Oct 30 18:54:39.269163 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.269117 4531 cpu_manager.go:414] "containerMap: removing container" podUID="57e07223-5536-45a0-931f-3c3874c63a67" containerName="revoke-keys" Oct 30 18:54:39.269163 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.269123 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57e07223-5536-45a0-931f-3c3874c63a67" containerName="revoke-keys" Oct 30 18:54:39.269266 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.269211 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="57e07223-5536-45a0-931f-3c3874c63a67" containerName="revoke-keys" Oct 30 18:54:39.269266 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.269222 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfb982b4-d179-4810-bbcd-e938c9797768" containerName="revoke-keys" Oct 30 18:54:39.273761 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.273742 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e5936-tvqhc" Oct 30 18:54:39.327465 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.327410 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:54:39.327793 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.327779 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:54:39.327892 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.327828 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:54:39.327958 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.327912 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:54:39.332064 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.332044 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-185e5936-tvqhc"] Oct 30 18:54:39.373688 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.373664 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6blh\" (UniqueName: \"kubernetes.io/projected/c4187bdc-ca1e-440c-97dd-2c9b97525a82-kube-api-access-d6blh\") pod \"maas-api-revoke-sub-185e5936-tvqhc\" (UID: \"c4187bdc-ca1e-440c-97dd-2c9b97525a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e5936-tvqhc" Oct 30 18:54:39.373799 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.373720 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4187bdc-ca1e-440c-97dd-2c9b97525a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-185e5936-tvqhc\" (UID: \"c4187bdc-ca1e-440c-97dd-2c9b97525a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e5936-tvqhc" Oct 30 18:54:39.373848 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.373802 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4187bdc-ca1e-440c-97dd-2c9b97525a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-185e5936-tvqhc\" (UID: \"c4187bdc-ca1e-440c-97dd-2c9b97525a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e5936-tvqhc" Oct 30 18:54:39.475395 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.475354 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6blh\" (UniqueName: \"kubernetes.io/projected/c4187bdc-ca1e-440c-97dd-2c9b97525a82-kube-api-access-d6blh\") pod \"maas-api-revoke-sub-185e5936-tvqhc\" (UID: \"c4187bdc-ca1e-440c-97dd-2c9b97525a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e5936-tvqhc" Oct 30 18:54:39.475709 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.475689 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4187bdc-ca1e-440c-97dd-2c9b97525a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-185e5936-tvqhc\" (UID: \"c4187bdc-ca1e-440c-97dd-2c9b97525a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e5936-tvqhc" Oct 30 18:54:39.475963 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.475937 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4187bdc-ca1e-440c-97dd-2c9b97525a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-185e5936-tvqhc\" (UID: \"c4187bdc-ca1e-440c-97dd-2c9b97525a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e5936-tvqhc" Oct 30 18:54:39.476893 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.476865 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4187bdc-ca1e-440c-97dd-2c9b97525a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-185e5936-tvqhc\" (UID: \"c4187bdc-ca1e-440c-97dd-2c9b97525a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e5936-tvqhc" Oct 30 18:54:39.479487 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.479462 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4187bdc-ca1e-440c-97dd-2c9b97525a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-185e5936-tvqhc\" (UID: \"c4187bdc-ca1e-440c-97dd-2c9b97525a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e5936-tvqhc" Oct 30 18:54:39.537033 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.536995 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6blh\" (UniqueName: \"kubernetes.io/projected/c4187bdc-ca1e-440c-97dd-2c9b97525a82-kube-api-access-d6blh\") pod \"maas-api-revoke-sub-185e5936-tvqhc\" (UID: \"c4187bdc-ca1e-440c-97dd-2c9b97525a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e5936-tvqhc" Oct 30 18:54:39.586907 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.586844 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e5936-tvqhc" Oct 30 18:54:39.790458 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.790428 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2db487af-nqs7s"] Oct 30 18:54:39.795215 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.795194 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db487af-nqs7s" Oct 30 18:54:39.853840 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.853780 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2db487af-nqs7s"] Oct 30 18:54:39.880701 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.880673 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b2b354-8b50-40dd-a140-354e8e36bc1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2db487af-nqs7s\" (UID: \"03b2b354-8b50-40dd-a140-354e8e36bc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db487af-nqs7s" Oct 30 18:54:39.880856 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.880806 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q4c8\" (UniqueName: \"kubernetes.io/projected/03b2b354-8b50-40dd-a140-354e8e36bc1a-kube-api-access-6q4c8\") pod \"maas-api-revoke-sub-2db487af-nqs7s\" (UID: \"03b2b354-8b50-40dd-a140-354e8e36bc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db487af-nqs7s" Oct 30 18:54:39.880971 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.880952 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b2b354-8b50-40dd-a140-354e8e36bc1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2db487af-nqs7s\" (UID: \"03b2b354-8b50-40dd-a140-354e8e36bc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db487af-nqs7s" Oct 30 18:54:39.982124 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.982094 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b2b354-8b50-40dd-a140-354e8e36bc1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2db487af-nqs7s\" (UID: \"03b2b354-8b50-40dd-a140-354e8e36bc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db487af-nqs7s" Oct 30 18:54:39.982283 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.982238 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q4c8\" (UniqueName: \"kubernetes.io/projected/03b2b354-8b50-40dd-a140-354e8e36bc1a-kube-api-access-6q4c8\") pod \"maas-api-revoke-sub-2db487af-nqs7s\" (UID: \"03b2b354-8b50-40dd-a140-354e8e36bc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db487af-nqs7s" Oct 30 18:54:39.982350 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.982335 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b2b354-8b50-40dd-a140-354e8e36bc1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2db487af-nqs7s\" (UID: \"03b2b354-8b50-40dd-a140-354e8e36bc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db487af-nqs7s" Oct 30 18:54:39.982884 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.982861 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b2b354-8b50-40dd-a140-354e8e36bc1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2db487af-nqs7s\" (UID: \"03b2b354-8b50-40dd-a140-354e8e36bc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db487af-nqs7s" Oct 30 18:54:39.984512 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:39.984495 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b2b354-8b50-40dd-a140-354e8e36bc1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2db487af-nqs7s\" (UID: \"03b2b354-8b50-40dd-a140-354e8e36bc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db487af-nqs7s" Oct 30 18:54:40.024834 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:40.024812 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-185e5936-tvqhc"] Oct 30 18:54:40.025736 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:54:40.025714 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4187bdc_ca1e_440c_97dd_2c9b97525a82.slice/crio-0b7efaa900e09c79c2d760927c1dcba8b26b51b07948d9e98358cf1f4fb0b130 WatchSource:0}: Error finding container 0b7efaa900e09c79c2d760927c1dcba8b26b51b07948d9e98358cf1f4fb0b130: Status 404 returned error can't find the container with id 0b7efaa900e09c79c2d760927c1dcba8b26b51b07948d9e98358cf1f4fb0b130 Oct 30 18:54:40.043909 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:40.043884 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q4c8\" (UniqueName: \"kubernetes.io/projected/03b2b354-8b50-40dd-a140-354e8e36bc1a-kube-api-access-6q4c8\") pod \"maas-api-revoke-sub-2db487af-nqs7s\" (UID: \"03b2b354-8b50-40dd-a140-354e8e36bc1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db487af-nqs7s" Oct 30 18:54:40.098790 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:40.098766 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c" path="/var/lib/kubelet/pods/fe1abf4c-f6c1-4b5d-a16b-ea1544d3fa9c/volumes" Oct 30 18:54:40.108770 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:40.108724 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db487af-nqs7s" Oct 30 18:54:40.486636 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:40.486556 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2db487af-nqs7s"] Oct 30 18:54:40.495612 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:54:40.495582 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b2b354_8b50_40dd_a140_354e8e36bc1a.slice/crio-2a856d0a87d147b4b341c3180d7099a1aa9c3504e907dead84f08ba40c465074 WatchSource:0}: Error finding container 2a856d0a87d147b4b341c3180d7099a1aa9c3504e907dead84f08ba40c465074: Status 404 returned error can't find the container with id 2a856d0a87d147b4b341c3180d7099a1aa9c3504e907dead84f08ba40c465074 Oct 30 18:54:40.659428 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:40.659394 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e5936-tvqhc" event={"ID":"c4187bdc-ca1e-440c-97dd-2c9b97525a82","Type":"ContainerStarted","Data":"6781778bbf00a6d79a11cfd00da67a11c6444e3edc0c0aaf86c2554e09b12275"} Oct 30 18:54:40.659592 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:40.659480 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e5936-tvqhc" event={"ID":"c4187bdc-ca1e-440c-97dd-2c9b97525a82","Type":"ContainerStarted","Data":"0b7efaa900e09c79c2d760927c1dcba8b26b51b07948d9e98358cf1f4fb0b130"} Oct 30 18:54:40.660943 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:40.660910 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db487af-nqs7s" event={"ID":"03b2b354-8b50-40dd-a140-354e8e36bc1a","Type":"ContainerStarted","Data":"2a856d0a87d147b4b341c3180d7099a1aa9c3504e907dead84f08ba40c465074"} Oct 30 18:54:40.777690 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:40.777635 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e5936-tvqhc" podStartSLOduration=1.7776219420000001 podStartE2EDuration="1.777621942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:40.775052215 +0000 UTC m=+1240.802157190" watchObservedRunningTime="2026-10-30 18:54:40.777621942 +0000 UTC m=+1240.804726901" Oct 30 18:54:41.668295 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:41.668225 4531 generic.go:356] "Generic (PLEG): container finished" podID="03b2b354-8b50-40dd-a140-354e8e36bc1a" containerID="cc613af989d7c13e5486361c0c41816081a131c176916c5a4ee3b2985ad8e8ba" exitCode=0 Oct 30 18:54:41.668681 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:41.668314 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db487af-nqs7s" event={"ID":"03b2b354-8b50-40dd-a140-354e8e36bc1a","Type":"ContainerDied","Data":"cc613af989d7c13e5486361c0c41816081a131c176916c5a4ee3b2985ad8e8ba"} Oct 30 18:54:41.670115 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:41.670090 4531 generic.go:356] "Generic (PLEG): container finished" podID="c4187bdc-ca1e-440c-97dd-2c9b97525a82" containerID="6781778bbf00a6d79a11cfd00da67a11c6444e3edc0c0aaf86c2554e09b12275" exitCode=0 Oct 30 18:54:41.670243 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:41.670166 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e5936-tvqhc" event={"ID":"c4187bdc-ca1e-440c-97dd-2c9b97525a82","Type":"ContainerDied","Data":"6781778bbf00a6d79a11cfd00da67a11c6444e3edc0c0aaf86c2554e09b12275"} Oct 30 18:54:42.876410 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:42.876382 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db487af-nqs7s" Oct 30 18:54:42.880532 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:42.880510 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e5936-tvqhc" Oct 30 18:54:43.020429 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:43.020396 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03b2b354-8b50-40dd-a140-354e8e36bc1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b2b354-8b50-40dd-a140-354e8e36bc1a-maas-api-xks-ca\") pod \"03b2b354-8b50-40dd-a140-354e8e36bc1a\" (UID: \"03b2b354-8b50-40dd-a140-354e8e36bc1a\") " Oct 30 18:54:43.020677 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:43.020434 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4187bdc-ca1e-440c-97dd-2c9b97525a82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4187bdc-ca1e-440c-97dd-2c9b97525a82-maas-api-openshift-ca\") pod \"c4187bdc-ca1e-440c-97dd-2c9b97525a82\" (UID: \"c4187bdc-ca1e-440c-97dd-2c9b97525a82\") " Oct 30 18:54:43.020677 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:43.020529 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03b2b354-8b50-40dd-a140-354e8e36bc1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b2b354-8b50-40dd-a140-354e8e36bc1a-maas-api-openshift-ca\") pod \"03b2b354-8b50-40dd-a140-354e8e36bc1a\" (UID: \"03b2b354-8b50-40dd-a140-354e8e36bc1a\") " Oct 30 18:54:43.020677 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:43.020582 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4187bdc-ca1e-440c-97dd-2c9b97525a82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4187bdc-ca1e-440c-97dd-2c9b97525a82-maas-api-xks-ca\") pod \"c4187bdc-ca1e-440c-97dd-2c9b97525a82\" (UID: \"c4187bdc-ca1e-440c-97dd-2c9b97525a82\") " Oct 30 18:54:43.020677 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:43.020646 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03b2b354-8b50-40dd-a140-354e8e36bc1a-kube-api-access-6q4c8\" (UniqueName: \"kubernetes.io/projected/03b2b354-8b50-40dd-a140-354e8e36bc1a-kube-api-access-6q4c8\") pod \"03b2b354-8b50-40dd-a140-354e8e36bc1a\" (UID: \"03b2b354-8b50-40dd-a140-354e8e36bc1a\") " Oct 30 18:54:43.020916 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:43.020678 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4187bdc-ca1e-440c-97dd-2c9b97525a82-kube-api-access-d6blh\" (UniqueName: \"kubernetes.io/projected/c4187bdc-ca1e-440c-97dd-2c9b97525a82-kube-api-access-d6blh\") pod \"c4187bdc-ca1e-440c-97dd-2c9b97525a82\" (UID: \"c4187bdc-ca1e-440c-97dd-2c9b97525a82\") " Oct 30 18:54:43.020916 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:43.020866 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4187bdc-ca1e-440c-97dd-2c9b97525a82-maas-api-openshift-ca" pod "c4187bdc-ca1e-440c-97dd-2c9b97525a82" (UID: "c4187bdc-ca1e-440c-97dd-2c9b97525a82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:43.021020 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:43.020951 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b2b354-8b50-40dd-a140-354e8e36bc1a-maas-api-openshift-ca" pod "03b2b354-8b50-40dd-a140-354e8e36bc1a" (UID: "03b2b354-8b50-40dd-a140-354e8e36bc1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:43.021260 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:43.021234 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4187bdc-ca1e-440c-97dd-2c9b97525a82-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:54:43.021536 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:43.021264 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b2b354-8b50-40dd-a140-354e8e36bc1a-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:54:43.023082 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:43.023059 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4187bdc-ca1e-440c-97dd-2c9b97525a82-kube-api-access-d6blh" pod "c4187bdc-ca1e-440c-97dd-2c9b97525a82" (UID: "c4187bdc-ca1e-440c-97dd-2c9b97525a82"). InnerVolumeSpecName "kube-api-access-d6blh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:43.023218 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:43.023077 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b2b354-8b50-40dd-a140-354e8e36bc1a-kube-api-access-6q4c8" pod "03b2b354-8b50-40dd-a140-354e8e36bc1a" (UID: "03b2b354-8b50-40dd-a140-354e8e36bc1a"). InnerVolumeSpecName "kube-api-access-6q4c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:43.023218 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:43.023122 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03b2b354-8b50-40dd-a140-354e8e36bc1a-maas-api-xks-ca" pod "03b2b354-8b50-40dd-a140-354e8e36bc1a" (UID: "03b2b354-8b50-40dd-a140-354e8e36bc1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:54:43.023664 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:43.023650 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4187bdc-ca1e-440c-97dd-2c9b97525a82-maas-api-xks-ca" pod "c4187bdc-ca1e-440c-97dd-2c9b97525a82" (UID: "c4187bdc-ca1e-440c-97dd-2c9b97525a82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:54:43.122758 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:43.122727 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b2b354-8b50-40dd-a140-354e8e36bc1a-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:54:43.122758 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:43.122758 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4187bdc-ca1e-440c-97dd-2c9b97525a82-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:54:43.122952 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:43.122773 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q4c8\" (UniqueName: \"kubernetes.io/projected/03b2b354-8b50-40dd-a140-354e8e36bc1a-kube-api-access-6q4c8\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:54:43.122952 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:43.122787 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6blh\" (UniqueName: \"kubernetes.io/projected/c4187bdc-ca1e-440c-97dd-2c9b97525a82-kube-api-access-d6blh\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:54:43.685699 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:43.685667 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e5936-tvqhc" Oct 30 18:54:43.685908 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:43.685706 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-185e5936-tvqhc" event={"ID":"c4187bdc-ca1e-440c-97dd-2c9b97525a82","Type":"ContainerDied","Data":"0b7efaa900e09c79c2d760927c1dcba8b26b51b07948d9e98358cf1f4fb0b130"} Oct 30 18:54:43.685908 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:43.685775 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b7efaa900e09c79c2d760927c1dcba8b26b51b07948d9e98358cf1f4fb0b130" Oct 30 18:54:43.687388 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:43.687365 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db487af-nqs7s" event={"ID":"03b2b354-8b50-40dd-a140-354e8e36bc1a","Type":"ContainerDied","Data":"2a856d0a87d147b4b341c3180d7099a1aa9c3504e907dead84f08ba40c465074"} Oct 30 18:54:43.687512 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:43.687402 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2db487af-nqs7s" Oct 30 18:54:43.687512 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:43.687409 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a856d0a87d147b4b341c3180d7099a1aa9c3504e907dead84f08ba40c465074" Oct 30 18:54:44.129934 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:44.129900 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a174c9f-2w24v"] Oct 30 18:54:44.130362 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:44.129945 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a174c9f-2w24v"] Oct 30 18:54:45.498156 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:45.497209 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" podUID="1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 30 18:54:45.606162 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:45.594205 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" podUID="471f1171-e71f-4da2-862d-5d232c45d6cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:54:45.606363 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:45.606214 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" podUID="586f370d-9916-43a9-ac7d-e77cc3fcb92f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 18:54:45.649155 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:45.642220 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" podUID="678454ff-4761-4bd4-b2a3-2b431bca7934" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 18:54:45.857211 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:45.855220 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" podUID="1a5f18bf-96ba-4edf-a758-206a1bd926a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:54:45.876260 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:45.876214 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" podUID="f23b88e6-d3b4-4975-818a-22cee9d30dce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:54:46.118498 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:46.118399 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbb7c4d1-e974-4b29-bca0-73f3791b378b" path="/var/lib/kubelet/pods/cbb7c4d1-e974-4b29-bca0-73f3791b378b/volumes" Oct 30 18:54:46.228275 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:46.228232 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" podUID="0679a60b-f854-49dd-a87d-3ea640cac995" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:54:46.508212 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:46.497721 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" podUID="1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:54:46.532674 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:46.531830 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 18:54:46.653155 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:46.629299 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" podUID="586f370d-9916-43a9-ac7d-e77cc3fcb92f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:54:46.653155 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:46.629761 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 18:54:46.658152 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:46.653507 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" podUID="678454ff-4761-4bd4-b2a3-2b431bca7934" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:54:46.658152 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:46.653947 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 18:54:46.885326 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:46.885203 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" podUID="f23b88e6-d3b4-4975-818a-22cee9d30dce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:54:46.911467 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:46.911411 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" podUID="1a5f18bf-96ba-4edf-a758-206a1bd926a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 30 18:54:46.912630 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:46.912594 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 18:54:46.913316 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:46.913302 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 18:54:46.936497 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:46.935588 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" podUID="5a868137-eb41-433c-ade6-58eb8c96d623" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:54:47.010060 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:47.009249 4531 patch_prober.go:29] interesting pod/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:54:47.010060 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:47.009333 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" podUID="6f0c5609-7aa9-4360-9818-6bdbc744554a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:54:47.252208 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:47.232371 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" podUID="0679a60b-f854-49dd-a87d-3ea640cac995" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:54:47.252208 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:47.232827 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 18:54:47.422422 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:47.421961 4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" podUID="586f370d-9916-43a9-ac7d-e77cc3fcb92f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:54:47.507208 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:47.503778 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" podUID="1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 30 18:54:47.532155 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:47.503718 4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" podUID="1a5f18bf-96ba-4edf-a758-206a1bd926a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": context deadline exceeded" Oct 30 18:54:47.594297 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:47.594255 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" podUID="471f1171-e71f-4da2-862d-5d232c45d6cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 18:54:47.637721 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:47.637670 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" podUID="586f370d-9916-43a9-ac7d-e77cc3fcb92f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 18:54:47.662008 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:47.655971 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" podUID="678454ff-4761-4bd4-b2a3-2b431bca7934" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 18:54:47.887373 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:47.887263 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" podUID="f23b88e6-d3b4-4975-818a-22cee9d30dce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:54:47.915035 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:47.914221 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" podUID="1a5f18bf-96ba-4edf-a758-206a1bd926a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:54:47.976434 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:47.944001 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k"] Oct 30 18:54:47.976434 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:47.944642 4531 cpu_manager.go:414] "containerMap: removing container" podUID="c4187bdc-ca1e-440c-97dd-2c9b97525a82" containerName="revoke-keys" Oct 30 18:54:47.976434 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:47.944659 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4187bdc-ca1e-440c-97dd-2c9b97525a82" containerName="revoke-keys" Oct 30 18:54:47.976434 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:47.944713 4531 cpu_manager.go:414] "containerMap: removing container" podUID="03b2b354-8b50-40dd-a140-354e8e36bc1a" containerName="revoke-keys" Oct 30 18:54:47.976434 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:47.944721 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03b2b354-8b50-40dd-a140-354e8e36bc1a" containerName="revoke-keys" Oct 30 18:54:47.976434 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:47.944815 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="03b2b354-8b50-40dd-a140-354e8e36bc1a" containerName="revoke-keys" Oct 30 18:54:47.976434 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:47.944828 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4187bdc-ca1e-440c-97dd-2c9b97525a82" containerName="revoke-keys" Oct 30 18:54:47.976434 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:47.975591 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" podUID="5a868137-eb41-433c-ade6-58eb8c96d623" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:54:48.059225 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:48.059181 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 18:54:48.060657 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:48.060638 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k" Oct 30 18:54:48.064291 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:48.064270 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k"] Oct 30 18:54:48.140311 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:48.140234 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-66baf45c\"" Oct 30 18:54:48.164164 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:48.163448 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-66baf45c-dockercfg-8c5j9\"" Oct 30 18:54:48.192245 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:48.192220 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 18:54:48.239872 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:48.239832 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" podUID="0679a60b-f854-49dd-a87d-3ea640cac995" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:54:48.280464 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:48.280427 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57tff\" (UniqueName: \"kubernetes.io/projected/0e569a12-fea9-4875-b068-15c7acbb0d3a-kube-api-access-57tff\") pod \"payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k\" (UID: \"0e569a12-fea9-4875-b068-15c7acbb0d3a\") " pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k" Oct 30 18:54:48.282145 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:48.282104 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e569a12-fea9-4875-b068-15c7acbb0d3a-plugins-config-volume\") pod \"payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k\" (UID: \"0e569a12-fea9-4875-b068-15c7acbb0d3a\") " pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k" Oct 30 18:54:48.310230 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:48.310207 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 18:54:48.313529 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:48.313508 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 18:54:48.402199 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:48.389031 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57tff\" (UniqueName: \"kubernetes.io/projected/0e569a12-fea9-4875-b068-15c7acbb0d3a-kube-api-access-57tff\") pod \"payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k\" (UID: \"0e569a12-fea9-4875-b068-15c7acbb0d3a\") " pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k" Oct 30 18:54:48.402199 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:48.389091 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e569a12-fea9-4875-b068-15c7acbb0d3a-plugins-config-volume\") pod \"payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k\" (UID: \"0e569a12-fea9-4875-b068-15c7acbb0d3a\") " pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k" Oct 30 18:54:48.402199 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:48.390049 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e569a12-fea9-4875-b068-15c7acbb0d3a-plugins-config-volume\") pod \"payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k\" (UID: \"0e569a12-fea9-4875-b068-15c7acbb0d3a\") " pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k" Oct 30 18:54:48.441154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:48.440399 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 18:54:48.510369 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:48.510331 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" podUID="1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 30 18:54:48.511866 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:48.511815 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 18:54:48.601424 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:48.601394 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57tff\" (UniqueName: \"kubernetes.io/projected/0e569a12-fea9-4875-b068-15c7acbb0d3a-kube-api-access-57tff\") pod \"payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k\" (UID: \"0e569a12-fea9-4875-b068-15c7acbb0d3a\") " pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k" Oct 30 18:54:48.601938 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:48.601605 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 18:54:48.604015 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:48.603984 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 18:54:48.785200 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:48.785168 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k" Oct 30 18:54:49.532579 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:49.532551 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k"] Oct 30 18:54:49.532714 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:54:49.532671 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e569a12_fea9_4875_b068_15c7acbb0d3a.slice/crio-382975a118e0cba4940db4f34f2020f3dae781475f8bb997c4f8189f76eff894 WatchSource:0}: Error finding container 382975a118e0cba4940db4f34f2020f3dae781475f8bb997c4f8189f76eff894: Status 404 returned error can't find the container with id 382975a118e0cba4940db4f34f2020f3dae781475f8bb997c4f8189f76eff894 Oct 30 18:54:50.119908 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:50.119867 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k" event={"ID":"0e569a12-fea9-4875-b068-15c7acbb0d3a","Type":"ContainerStarted","Data":"b96862750592350ecad25af7037710d517badcd054558b4ed551863a2aefd225"} Oct 30 18:54:50.120352 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:50.120035 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k" Oct 30 18:54:50.120352 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:50.120096 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k" event={"ID":"0e569a12-fea9-4875-b068-15c7acbb0d3a","Type":"ContainerStarted","Data":"382975a118e0cba4940db4f34f2020f3dae781475f8bb997c4f8189f76eff894"} Oct 30 18:54:50.238922 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:50.238872 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k" podStartSLOduration=3.2388589469999998 podStartE2EDuration="3.238858947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:50.23787354 +0000 UTC m=+1250.264978503" watchObservedRunningTime="2026-10-30 18:54:50.238858947 +0000 UTC m=+1250.265963906" Oct 30 18:54:56.051433 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:56.051394 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08dc48c-s8w9l"] Oct 30 18:54:56.100381 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:56.100353 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08dc48c-s8w9l"] Oct 30 18:54:56.398406 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:54:56.398321 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:54:58.098668 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:54:58.098634 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="935bde25-694f-43fb-9874-91a4e5710360" path="/var/lib/kubelet/pods/935bde25-694f-43fb-9874-91a4e5710360/volumes" Oct 30 18:55:01.128978 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:01.128941 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k" Oct 30 18:55:08.467844 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.467796 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779"] Oct 30 18:55:08.473389 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.473353 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.578981 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.578940 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779"] Oct 30 18:55:08.615675 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.615635 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-data\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.615911 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.615706 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-workload-socket\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.615911 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.615790 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-workload-certs\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.615911 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.615830 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-token\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.616075 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.616023 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-envoy\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.616151 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.616068 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrrct\" (UniqueName: \"kubernetes.io/projected/26466126-8cc0-4784-9c40-a2d16bc3256d-kube-api-access-xrrct\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.616231 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.616214 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26466126-8cc0-4784-9c40-a2d16bc3256d-istiod-ca-cert\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.616288 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.616241 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-credential-socket\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.616288 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.616259 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-podinfo\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.717913 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.717811 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26466126-8cc0-4784-9c40-a2d16bc3256d-istiod-ca-cert\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.717913 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.717871 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-credential-socket\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.717913 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.717898 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-podinfo\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.718251 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.717966 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-data\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.718251 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.718038 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-workload-socket\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.718251 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.718076 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-workload-certs\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.718251 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.718176 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-token\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.718474 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.718323 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-envoy\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.718474 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.718358 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrrct\" (UniqueName: \"kubernetes.io/projected/26466126-8cc0-4784-9c40-a2d16bc3256d-kube-api-access-xrrct\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.718474 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.718450 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-credential-socket\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.718474 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.718465 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-data\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.719208 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.719181 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-workload-certs\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.719399 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.719262 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26466126-8cc0-4784-9c40-a2d16bc3256d-istiod-ca-cert\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.719565 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.719441 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-workload-socket\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.721782 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.721742 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-podinfo\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.722115 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.722090 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-envoy\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.779991 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.779952 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-token\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.780224 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.780023 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrrct\" (UniqueName: \"kubernetes.io/projected/26466126-8cc0-4784-9c40-a2d16bc3256d-kube-api-access-xrrct\") pod \"e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:08.787975 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:08.787932 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:09.227176 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:09.225455 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779"] Oct 30 18:55:09.227176 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:55:09.226708 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26466126_8cc0_4784_9c40_a2d16bc3256d.slice/crio-413a6057ef03cee8481b066e3ba86b860b4c07629ec226661ec90b07d70229d1 WatchSource:0}: Error finding container 413a6057ef03cee8481b066e3ba86b860b4c07629ec226661ec90b07d70229d1: Status 404 returned error can't find the container with id 413a6057ef03cee8481b066e3ba86b860b4c07629ec226661ec90b07d70229d1 Oct 30 18:55:09.229634 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:09.229589 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:55:09.229759 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:09.229694 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:55:09.229759 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:09.229746 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:55:09.240311 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:09.240269 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" event={"ID":"26466126-8cc0-4784-9c40-a2d16bc3256d","Type":"ContainerStarted","Data":"413a6057ef03cee8481b066e3ba86b860b4c07629ec226661ec90b07d70229d1"} Oct 30 18:55:10.251888 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:10.251843 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" event={"ID":"26466126-8cc0-4784-9c40-a2d16bc3256d","Type":"ContainerStarted","Data":"6f425a94e5c2ae0ef2a8eff0adf72cc34402be61053ee1b9d85e4c3908d95676"} Oct 30 18:55:10.441535 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:10.440070 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" podStartSLOduration=2.440043598 podStartE2EDuration="2.440043598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:10.425411569 +0000 UTC m=+1270.452516530" watchObservedRunningTime="2026-10-30 18:55:10.440043598 +0000 UTC m=+1270.467148556" Oct 30 18:55:10.789098 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:10.789054 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:10.790974 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:10.790947 4531 patch_prober.go:29] interesting pod/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:55:10.791054 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:10.791002 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" podUID="26466126-8cc0-4784-9c40-a2d16bc3256d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 18:55:11.788844 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:11.788799 4531 patch_prober.go:29] interesting pod/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:55:11.789282 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:11.788859 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" podUID="26466126-8cc0-4784-9c40-a2d16bc3256d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 18:55:12.788941 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:12.788900 4531 patch_prober.go:29] interesting pod/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:55:12.789320 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:12.788966 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" podUID="26466126-8cc0-4784-9c40-a2d16bc3256d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 18:55:13.788739 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:13.788701 4531 patch_prober.go:29] interesting pod/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:55:13.788901 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:13.788769 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" podUID="26466126-8cc0-4784-9c40-a2d16bc3256d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 18:55:14.788803 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:14.788768 4531 patch_prober.go:29] interesting pod/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:55:14.789316 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:14.788824 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" podUID="26466126-8cc0-4784-9c40-a2d16bc3256d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 18:55:15.789050 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:15.789011 4531 patch_prober.go:29] interesting pod/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:55:15.789518 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:15.789091 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" podUID="26466126-8cc0-4784-9c40-a2d16bc3256d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 18:55:16.793835 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:16.793797 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:16.794374 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:16.794241 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:16.795314 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:16.795291 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 18:55:19.050234 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:19.050187 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba9d1ff8-dhrxj"] Oct 30 18:55:19.099291 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:19.099251 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba9d1ff8-dhrxj"] Oct 30 18:55:20.102925 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:20.102884 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00d659b2-ee5c-43e9-a2e5-82bbbee24f0c" path="/var/lib/kubelet/pods/00d659b2-ee5c-43e9-a2e5-82bbbee24f0c/volumes" Oct 30 18:55:26.400653 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:55:26.400605 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:55:40.999346 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:40.999264 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ef8cbc-vmgh7"] Oct 30 18:55:41.004513 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:41.004486 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ef8cbc-vmgh7" Oct 30 18:55:41.039774 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:41.039729 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9lp2\" (UniqueName: \"kubernetes.io/projected/97fa301a-7f41-4d7c-8ba4-f67ad32f256a-kube-api-access-x9lp2\") pod \"maas-api-revoke-sub-e1ef8cbc-vmgh7\" (UID: \"97fa301a-7f41-4d7c-8ba4-f67ad32f256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ef8cbc-vmgh7" Oct 30 18:55:41.039958 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:41.039867 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97fa301a-7f41-4d7c-8ba4-f67ad32f256a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1ef8cbc-vmgh7\" (UID: \"97fa301a-7f41-4d7c-8ba4-f67ad32f256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ef8cbc-vmgh7" Oct 30 18:55:41.040001 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:41.039965 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97fa301a-7f41-4d7c-8ba4-f67ad32f256a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1ef8cbc-vmgh7\" (UID: \"97fa301a-7f41-4d7c-8ba4-f67ad32f256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ef8cbc-vmgh7" Oct 30 18:55:41.057724 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:41.057683 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:55:41.061867 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:41.061829 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ef8cbc-vmgh7"] Oct 30 18:55:41.107191 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:41.107156 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:55:41.107191 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:41.107169 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:55:41.107191 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:41.107187 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:55:41.141044 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:41.140981 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97fa301a-7f41-4d7c-8ba4-f67ad32f256a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1ef8cbc-vmgh7\" (UID: \"97fa301a-7f41-4d7c-8ba4-f67ad32f256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ef8cbc-vmgh7" Oct 30 18:55:41.141292 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:41.141159 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97fa301a-7f41-4d7c-8ba4-f67ad32f256a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1ef8cbc-vmgh7\" (UID: \"97fa301a-7f41-4d7c-8ba4-f67ad32f256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ef8cbc-vmgh7" Oct 30 18:55:41.141292 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:41.141271 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9lp2\" (UniqueName: \"kubernetes.io/projected/97fa301a-7f41-4d7c-8ba4-f67ad32f256a-kube-api-access-x9lp2\") pod \"maas-api-revoke-sub-e1ef8cbc-vmgh7\" (UID: \"97fa301a-7f41-4d7c-8ba4-f67ad32f256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ef8cbc-vmgh7" Oct 30 18:55:41.141957 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:41.141863 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97fa301a-7f41-4d7c-8ba4-f67ad32f256a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1ef8cbc-vmgh7\" (UID: \"97fa301a-7f41-4d7c-8ba4-f67ad32f256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ef8cbc-vmgh7" Oct 30 18:55:41.143764 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:41.143740 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97fa301a-7f41-4d7c-8ba4-f67ad32f256a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1ef8cbc-vmgh7\" (UID: \"97fa301a-7f41-4d7c-8ba4-f67ad32f256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ef8cbc-vmgh7" Oct 30 18:55:41.202686 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:41.202647 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9lp2\" (UniqueName: \"kubernetes.io/projected/97fa301a-7f41-4d7c-8ba4-f67ad32f256a-kube-api-access-x9lp2\") pod \"maas-api-revoke-sub-e1ef8cbc-vmgh7\" (UID: \"97fa301a-7f41-4d7c-8ba4-f67ad32f256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ef8cbc-vmgh7" Oct 30 18:55:41.319319 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:41.319201 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ef8cbc-vmgh7" Oct 30 18:55:41.703009 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:41.702969 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ef8cbc-vmgh7"] Oct 30 18:55:41.704532 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:55:41.704490 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97fa301a_7f41_4d7c_8ba4_f67ad32f256a.slice/crio-ffd83d2ce8a871ab55cc41f9064bafc3c494200cd11fb9b8c6fae1afa7ef2bb4 WatchSource:0}: Error finding container ffd83d2ce8a871ab55cc41f9064bafc3c494200cd11fb9b8c6fae1afa7ef2bb4: Status 404 returned error can't find the container with id ffd83d2ce8a871ab55cc41f9064bafc3c494200cd11fb9b8c6fae1afa7ef2bb4 Oct 30 18:55:42.544077 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:42.544037 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ef8cbc-vmgh7" event={"ID":"97fa301a-7f41-4d7c-8ba4-f67ad32f256a","Type":"ContainerStarted","Data":"c30cbde9692305c53435962479be92fa3329f6fe40981cd770e586820bbf75c3"} Oct 30 18:55:42.544527 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:42.544105 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ef8cbc-vmgh7" event={"ID":"97fa301a-7f41-4d7c-8ba4-f67ad32f256a","Type":"ContainerStarted","Data":"ffd83d2ce8a871ab55cc41f9064bafc3c494200cd11fb9b8c6fae1afa7ef2bb4"} Oct 30 18:55:42.664036 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:42.663980 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ef8cbc-vmgh7" podStartSLOduration=2.663964536 podStartE2EDuration="2.663964536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:42.659717822 +0000 UTC m=+1302.686822782" watchObservedRunningTime="2026-10-30 18:55:42.663964536 +0000 UTC m=+1302.691069494" Oct 30 18:55:43.552796 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:43.552758 4531 generic.go:356] "Generic (PLEG): container finished" podID="97fa301a-7f41-4d7c-8ba4-f67ad32f256a" containerID="c30cbde9692305c53435962479be92fa3329f6fe40981cd770e586820bbf75c3" exitCode=0 Oct 30 18:55:43.553273 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:43.552854 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ef8cbc-vmgh7" event={"ID":"97fa301a-7f41-4d7c-8ba4-f67ad32f256a","Type":"ContainerDied","Data":"c30cbde9692305c53435962479be92fa3329f6fe40981cd770e586820bbf75c3"} Oct 30 18:55:44.722869 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:44.722838 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ef8cbc-vmgh7" Oct 30 18:55:44.784580 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:44.784537 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97fa301a-7f41-4d7c-8ba4-f67ad32f256a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97fa301a-7f41-4d7c-8ba4-f67ad32f256a-maas-api-xks-ca\") pod \"97fa301a-7f41-4d7c-8ba4-f67ad32f256a\" (UID: \"97fa301a-7f41-4d7c-8ba4-f67ad32f256a\") " Oct 30 18:55:44.784766 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:44.784596 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97fa301a-7f41-4d7c-8ba4-f67ad32f256a-kube-api-access-x9lp2\" (UniqueName: \"kubernetes.io/projected/97fa301a-7f41-4d7c-8ba4-f67ad32f256a-kube-api-access-x9lp2\") pod \"97fa301a-7f41-4d7c-8ba4-f67ad32f256a\" (UID: \"97fa301a-7f41-4d7c-8ba4-f67ad32f256a\") " Oct 30 18:55:44.784766 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:44.784668 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97fa301a-7f41-4d7c-8ba4-f67ad32f256a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97fa301a-7f41-4d7c-8ba4-f67ad32f256a-maas-api-openshift-ca\") pod \"97fa301a-7f41-4d7c-8ba4-f67ad32f256a\" (UID: \"97fa301a-7f41-4d7c-8ba4-f67ad32f256a\") " Oct 30 18:55:44.785083 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:44.785059 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97fa301a-7f41-4d7c-8ba4-f67ad32f256a-maas-api-openshift-ca" pod "97fa301a-7f41-4d7c-8ba4-f67ad32f256a" (UID: "97fa301a-7f41-4d7c-8ba4-f67ad32f256a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:55:44.786969 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:44.786941 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97fa301a-7f41-4d7c-8ba4-f67ad32f256a-kube-api-access-x9lp2" pod "97fa301a-7f41-4d7c-8ba4-f67ad32f256a" (UID: "97fa301a-7f41-4d7c-8ba4-f67ad32f256a"). InnerVolumeSpecName "kube-api-access-x9lp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:55:44.786969 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:44.786954 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97fa301a-7f41-4d7c-8ba4-f67ad32f256a-maas-api-xks-ca" pod "97fa301a-7f41-4d7c-8ba4-f67ad32f256a" (UID: "97fa301a-7f41-4d7c-8ba4-f67ad32f256a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:55:44.886032 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:44.885924 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97fa301a-7f41-4d7c-8ba4-f67ad32f256a-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:55:44.886032 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:44.885965 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9lp2\" (UniqueName: \"kubernetes.io/projected/97fa301a-7f41-4d7c-8ba4-f67ad32f256a-kube-api-access-x9lp2\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:55:44.886032 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:44.885981 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97fa301a-7f41-4d7c-8ba4-f67ad32f256a-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:55:45.567910 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:45.567881 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ef8cbc-vmgh7" Oct 30 18:55:45.568090 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:45.567916 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ef8cbc-vmgh7" event={"ID":"97fa301a-7f41-4d7c-8ba4-f67ad32f256a","Type":"ContainerDied","Data":"ffd83d2ce8a871ab55cc41f9064bafc3c494200cd11fb9b8c6fae1afa7ef2bb4"} Oct 30 18:55:45.568090 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:45.567985 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffd83d2ce8a871ab55cc41f9064bafc3c494200cd11fb9b8c6fae1afa7ef2bb4" Oct 30 18:55:46.538539 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.538491 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn"] Oct 30 18:55:46.539038 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.538965 4531 cpu_manager.go:414] "containerMap: removing container" podUID="97fa301a-7f41-4d7c-8ba4-f67ad32f256a" containerName="revoke-keys" Oct 30 18:55:46.539038 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.538977 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97fa301a-7f41-4d7c-8ba4-f67ad32f256a" containerName="revoke-keys" Oct 30 18:55:46.539111 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.539060 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="97fa301a-7f41-4d7c-8ba4-f67ad32f256a" containerName="revoke-keys" Oct 30 18:55:46.542466 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.542440 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" Oct 30 18:55:46.605891 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.605854 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7zsj\" (UniqueName: \"kubernetes.io/projected/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-kube-api-access-p7zsj\") pod \"distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn\" (UID: \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" Oct 30 18:55:46.606105 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.605998 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-model-cache\") pod \"distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn\" (UID: \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" Oct 30 18:55:46.606105 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.606072 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-tmp-dir\") pod \"distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn\" (UID: \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" Oct 30 18:55:46.606319 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.606169 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-tls-certs\") pod \"distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn\" (UID: \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" Oct 30 18:55:46.606319 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.606201 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-home\") pod \"distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn\" (UID: \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" Oct 30 18:55:46.606319 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.606225 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-dshm\") pod \"distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn\" (UID: \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" Oct 30 18:55:46.644772 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.644732 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2ade09\"/\"distinct-2-w0-2ade09-kserve-self-signed-certs\"" Oct 30 18:55:46.648889 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.648850 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn"] Oct 30 18:55:46.652495 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.652449 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49"] Oct 30 18:55:46.656568 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.656543 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.707230 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.707172 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-token\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.707449 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.707268 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-podinfo\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.707449 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.707313 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-credential-socket\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.707449 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.707342 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-workload-certs\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.707449 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.707385 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7zsj\" (UniqueName: \"kubernetes.io/projected/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-kube-api-access-p7zsj\") pod \"distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn\" (UID: \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" Oct 30 18:55:46.709638 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.708999 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-4a7a52-openshift-default-dockercfg-9p592\"" Oct 30 18:55:46.709638 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.709251 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-envoy\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.709638 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.709322 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-model-cache\") pod \"distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn\" (UID: \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" Oct 30 18:55:46.709638 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.709379 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-data\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.709638 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.709408 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istiod-ca-cert\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.709638 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.709460 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-tmp-dir\") pod \"distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn\" (UID: \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" Oct 30 18:55:46.709638 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.709487 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn9mh\" (UniqueName: \"kubernetes.io/projected/f9341cd2-23c8-4c4e-a895-2529c03dad3c-kube-api-access-vn9mh\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.709638 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.709517 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-workload-socket\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.709638 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.709547 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-tls-certs\") pod \"distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn\" (UID: \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" Oct 30 18:55:46.709638 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.709573 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-home\") pod \"distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn\" (UID: \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" Oct 30 18:55:46.709638 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.709592 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-dshm\") pod \"distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn\" (UID: \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" Oct 30 18:55:46.710364 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.710277 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-tmp-dir\") pod \"distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn\" (UID: \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" Oct 30 18:55:46.710499 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.710478 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-model-cache\") pod \"distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn\" (UID: \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" Oct 30 18:55:46.711025 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.711004 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-home\") pod \"distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn\" (UID: \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" Oct 30 18:55:46.712304 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.712270 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-dshm\") pod \"distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn\" (UID: \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" Oct 30 18:55:46.713092 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.713071 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-tls-certs\") pod \"distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn\" (UID: \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" Oct 30 18:55:46.721224 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.721165 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49"] Oct 30 18:55:46.770655 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.770619 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7zsj\" (UniqueName: \"kubernetes.io/projected/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-kube-api-access-p7zsj\") pod \"distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn\" (UID: \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\") " pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" Oct 30 18:55:46.811341 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.811225 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-token\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.811341 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.811274 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-podinfo\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.811341 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.811299 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-credential-socket\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.811341 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.811315 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-workload-certs\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.811702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.811366 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-envoy\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.811702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.811430 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-data\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.811702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.811466 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istiod-ca-cert\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.811702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.811520 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn9mh\" (UniqueName: \"kubernetes.io/projected/f9341cd2-23c8-4c4e-a895-2529c03dad3c-kube-api-access-vn9mh\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.811702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.811549 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-workload-socket\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.811965 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.811757 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-credential-socket\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.811965 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.811947 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-workload-socket\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.812073 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.811983 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-workload-certs\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.812185 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.812164 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-data\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.812339 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.812319 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istiod-ca-cert\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.814493 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.814455 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-envoy\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.814754 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.814732 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-podinfo\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.855617 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.855571 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" Oct 30 18:55:46.875467 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.874553 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn9mh\" (UniqueName: \"kubernetes.io/projected/f9341cd2-23c8-4c4e-a895-2529c03dad3c-kube-api-access-vn9mh\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.886974 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.886933 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-token\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:46.989088 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:46.988063 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:47.303007 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:47.302973 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn"] Oct 30 18:55:47.303518 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:55:47.303496 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9dd3f9ea_88e6_4136_9ff9_ad64e6e3104e.slice/crio-fed1865ad8d83b9935356521d7acc0446c2074565fccb3221caeb24b34cfdd6a WatchSource:0}: Error finding container fed1865ad8d83b9935356521d7acc0446c2074565fccb3221caeb24b34cfdd6a: Status 404 returned error can't find the container with id fed1865ad8d83b9935356521d7acc0446c2074565fccb3221caeb24b34cfdd6a Oct 30 18:55:47.434521 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:47.434486 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49"] Oct 30 18:55:47.435606 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:55:47.435566 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9341cd2_23c8_4c4e_a895_2529c03dad3c.slice/crio-0ec8330f7cd2067c970b12bfc3765a841e9efdb44870fe4c11a5a3e54bfc31f4 WatchSource:0}: Error finding container 0ec8330f7cd2067c970b12bfc3765a841e9efdb44870fe4c11a5a3e54bfc31f4: Status 404 returned error can't find the container with id 0ec8330f7cd2067c970b12bfc3765a841e9efdb44870fe4c11a5a3e54bfc31f4 Oct 30 18:55:47.438419 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:47.438383 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:55:47.438514 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:47.438457 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:55:47.438514 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:47.438487 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:55:47.589368 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:47.589317 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" event={"ID":"f9341cd2-23c8-4c4e-a895-2529c03dad3c","Type":"ContainerStarted","Data":"16bc92296e0f13c8d46eeafbaede39fda7bbbe57c587621ea1b559bb901bf525"} Oct 30 18:55:47.589868 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:47.589383 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" event={"ID":"f9341cd2-23c8-4c4e-a895-2529c03dad3c","Type":"ContainerStarted","Data":"0ec8330f7cd2067c970b12bfc3765a841e9efdb44870fe4c11a5a3e54bfc31f4"} Oct 30 18:55:47.592318 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:47.592285 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" event={"ID":"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e","Type":"ContainerStarted","Data":"db5277d35bda9ea46c09e1c3dadcca354b37eff8d59a82ace86ec474d2e42c66"} Oct 30 18:55:47.592521 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:47.592325 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" event={"ID":"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e","Type":"ContainerStarted","Data":"fed1865ad8d83b9935356521d7acc0446c2074565fccb3221caeb24b34cfdd6a"} Oct 30 18:55:47.711991 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:47.711843 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" podStartSLOduration=1.711819932 podStartE2EDuration="1.711819932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:47.708746553 +0000 UTC m=+1307.735851550" watchObservedRunningTime="2026-10-30 18:55:47.711819932 +0000 UTC m=+1307.738924892" Oct 30 18:55:47.825189 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:47.825086 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" podStartSLOduration=1.825069142 podStartE2EDuration="1.825069142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:47.824489908 +0000 UTC m=+1307.851594868" watchObservedRunningTime="2026-10-30 18:55:47.825069142 +0000 UTC m=+1307.852174095" Oct 30 18:55:47.856522 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:47.856480 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" Oct 30 18:55:47.990394 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:47.990237 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:48.991367 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:48.991321 4531 patch_prober.go:29] interesting pod/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 18:55:48.991844 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:48.991396 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" podUID="f9341cd2-23c8-4c4e-a895-2529c03dad3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 18:55:49.989472 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:49.989430 4531 patch_prober.go:29] interesting pod/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 18:55:49.989673 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:49.989497 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" podUID="f9341cd2-23c8-4c4e-a895-2529c03dad3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 18:55:50.988792 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:50.988749 4531 patch_prober.go:29] interesting pod/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 18:55:50.989412 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:50.989374 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" podUID="f9341cd2-23c8-4c4e-a895-2529c03dad3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 18:55:51.988829 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:51.988794 4531 patch_prober.go:29] interesting pod/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 18:55:51.989318 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:51.988854 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" podUID="f9341cd2-23c8-4c4e-a895-2529c03dad3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 18:55:52.989047 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:52.989005 4531 patch_prober.go:29] interesting pod/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 18:55:52.989654 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:52.989064 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" podUID="f9341cd2-23c8-4c4e-a895-2529c03dad3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 18:55:53.989022 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:53.988982 4531 patch_prober.go:29] interesting pod/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 18:55:53.989232 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:53.989037 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" podUID="f9341cd2-23c8-4c4e-a895-2529c03dad3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 18:55:54.993472 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:54.993436 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:54.993916 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:54.993897 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:54.995005 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:54.994982 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 18:55:56.398083 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:55:56.398044 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:55:56.771431 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:56.771394 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac7496fa-jbqbv"] Oct 30 18:55:56.775476 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:56.775454 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7496fa-jbqbv" Oct 30 18:55:56.841671 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:56.841614 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:55:56.841846 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:56.841686 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:55:56.845780 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:56.845744 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac7496fa-jbqbv"] Oct 30 18:55:56.853533 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:56.853495 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/151d129d-222e-4c7b-91f1-828d24137732-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac7496fa-jbqbv\" (UID: \"151d129d-222e-4c7b-91f1-828d24137732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7496fa-jbqbv" Oct 30 18:55:56.853717 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:56.853562 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/151d129d-222e-4c7b-91f1-828d24137732-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac7496fa-jbqbv\" (UID: \"151d129d-222e-4c7b-91f1-828d24137732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7496fa-jbqbv" Oct 30 18:55:56.853822 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:56.853797 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blb7t\" (UniqueName: \"kubernetes.io/projected/151d129d-222e-4c7b-91f1-828d24137732-kube-api-access-blb7t\") pod \"maas-api-revoke-sub-ac7496fa-jbqbv\" (UID: \"151d129d-222e-4c7b-91f1-828d24137732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7496fa-jbqbv" Oct 30 18:55:56.856067 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:56.856040 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" Oct 30 18:55:56.869077 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:56.869051 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" Oct 30 18:55:56.890826 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:56.890792 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:55:56.890826 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:56.890813 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:55:56.955458 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:56.955415 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blb7t\" (UniqueName: \"kubernetes.io/projected/151d129d-222e-4c7b-91f1-828d24137732-kube-api-access-blb7t\") pod \"maas-api-revoke-sub-ac7496fa-jbqbv\" (UID: \"151d129d-222e-4c7b-91f1-828d24137732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7496fa-jbqbv" Oct 30 18:55:56.955685 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:56.955527 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/151d129d-222e-4c7b-91f1-828d24137732-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac7496fa-jbqbv\" (UID: \"151d129d-222e-4c7b-91f1-828d24137732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7496fa-jbqbv" Oct 30 18:55:56.955685 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:56.955579 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/151d129d-222e-4c7b-91f1-828d24137732-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac7496fa-jbqbv\" (UID: \"151d129d-222e-4c7b-91f1-828d24137732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7496fa-jbqbv" Oct 30 18:55:56.956314 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:56.956293 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/151d129d-222e-4c7b-91f1-828d24137732-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac7496fa-jbqbv\" (UID: \"151d129d-222e-4c7b-91f1-828d24137732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7496fa-jbqbv" Oct 30 18:55:56.958169 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:56.958146 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/151d129d-222e-4c7b-91f1-828d24137732-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac7496fa-jbqbv\" (UID: \"151d129d-222e-4c7b-91f1-828d24137732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7496fa-jbqbv" Oct 30 18:55:57.016517 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:57.016481 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blb7t\" (UniqueName: \"kubernetes.io/projected/151d129d-222e-4c7b-91f1-828d24137732-kube-api-access-blb7t\") pod \"maas-api-revoke-sub-ac7496fa-jbqbv\" (UID: \"151d129d-222e-4c7b-91f1-828d24137732\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7496fa-jbqbv" Oct 30 18:55:57.089975 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:57.089893 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7496fa-jbqbv" Oct 30 18:55:57.469050 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:57.469009 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac7496fa-jbqbv"] Oct 30 18:55:57.469873 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:55:57.469845 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151d129d_222e_4c7b_91f1_828d24137732.slice/crio-61f25af9af5c120e593a428bc0ac81ca58a2bf903bb4cda76046d024a5891fa8 WatchSource:0}: Error finding container 61f25af9af5c120e593a428bc0ac81ca58a2bf903bb4cda76046d024a5891fa8: Status 404 returned error can't find the container with id 61f25af9af5c120e593a428bc0ac81ca58a2bf903bb4cda76046d024a5891fa8 Oct 30 18:55:57.691648 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:57.691610 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7496fa-jbqbv" event={"ID":"151d129d-222e-4c7b-91f1-828d24137732","Type":"ContainerStarted","Data":"61f25af9af5c120e593a428bc0ac81ca58a2bf903bb4cda76046d024a5891fa8"} Oct 30 18:55:57.703512 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:57.703479 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" Oct 30 18:55:58.704353 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:58.704262 4531 generic.go:356] "Generic (PLEG): container finished" podID="151d129d-222e-4c7b-91f1-828d24137732" containerID="ed98cd01a1f723fbf1838c96276a843441baf00f7e0308f0aa9c6eb73824a249" exitCode=0 Oct 30 18:55:58.704353 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:58.704302 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7496fa-jbqbv" event={"ID":"151d129d-222e-4c7b-91f1-828d24137732","Type":"ContainerDied","Data":"ed98cd01a1f723fbf1838c96276a843441baf00f7e0308f0aa9c6eb73824a249"} Oct 30 18:55:59.891071 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:59.891037 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7496fa-jbqbv" Oct 30 18:55:59.996976 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:59.996942 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151d129d-222e-4c7b-91f1-828d24137732-kube-api-access-blb7t\" (UniqueName: \"kubernetes.io/projected/151d129d-222e-4c7b-91f1-828d24137732-kube-api-access-blb7t\") pod \"151d129d-222e-4c7b-91f1-828d24137732\" (UID: \"151d129d-222e-4c7b-91f1-828d24137732\") " Oct 30 18:55:59.997181 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:59.997057 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/151d129d-222e-4c7b-91f1-828d24137732-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/151d129d-222e-4c7b-91f1-828d24137732-maas-api-openshift-ca\") pod \"151d129d-222e-4c7b-91f1-828d24137732\" (UID: \"151d129d-222e-4c7b-91f1-828d24137732\") " Oct 30 18:55:59.997181 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:59.997093 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/151d129d-222e-4c7b-91f1-828d24137732-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/151d129d-222e-4c7b-91f1-828d24137732-maas-api-xks-ca\") pod \"151d129d-222e-4c7b-91f1-828d24137732\" (UID: \"151d129d-222e-4c7b-91f1-828d24137732\") " Oct 30 18:55:59.997516 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:59.997475 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/151d129d-222e-4c7b-91f1-828d24137732-maas-api-openshift-ca" pod "151d129d-222e-4c7b-91f1-828d24137732" (UID: "151d129d-222e-4c7b-91f1-828d24137732"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:55:59.999486 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:59.999458 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151d129d-222e-4c7b-91f1-828d24137732-kube-api-access-blb7t" pod "151d129d-222e-4c7b-91f1-828d24137732" (UID: "151d129d-222e-4c7b-91f1-828d24137732"). InnerVolumeSpecName "kube-api-access-blb7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:55:59.999486 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:55:59.999457 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/151d129d-222e-4c7b-91f1-828d24137732-maas-api-xks-ca" pod "151d129d-222e-4c7b-91f1-828d24137732" (UID: "151d129d-222e-4c7b-91f1-828d24137732"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:00.098151 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:00.098107 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blb7t\" (UniqueName: \"kubernetes.io/projected/151d129d-222e-4c7b-91f1-828d24137732-kube-api-access-blb7t\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:56:00.098151 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:00.098149 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/151d129d-222e-4c7b-91f1-828d24137732-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:56:00.098361 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:00.098161 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/151d129d-222e-4c7b-91f1-828d24137732-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:56:00.309967 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:00.309886 4531 scope.go:122] "RemoveContainer" containerID="cf5456e5a640bf5d601e963edda4c401744e1495b0363219a00ee543dfc3c007" Oct 30 18:56:00.321712 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:00.321686 4531 scope.go:122] "RemoveContainer" containerID="47cac700bdb5990f119e2dc3a0feb16e63189bf81c974737694f00f2f29ff232" Oct 30 18:56:00.335318 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:00.335291 4531 scope.go:122] "RemoveContainer" containerID="062c70f66bed8015f48450f6839ab1543ff882a566ecf83322f4fb933122693f" Oct 30 18:56:00.345400 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:00.345374 4531 scope.go:122] "RemoveContainer" containerID="4d88e5607b1c13d224401e9890e27570443ee0a4ec428487bcdbde83c02647ed" Oct 30 18:56:00.354973 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:00.354952 4531 scope.go:122] "RemoveContainer" containerID="922ef74e2aead53317444145cab344d77df5fad113c4df01b346e0a5391c4758" Oct 30 18:56:00.364799 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:00.364773 4531 scope.go:122] "RemoveContainer" containerID="cc44cb9a84286dba1efa1680d8f9fa22f22b2a9fb5b981e36694658b2edd25db" Oct 30 18:56:00.374253 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:00.374232 4531 scope.go:122] "RemoveContainer" containerID="b51402c0984199d9255490130dc91c46b9eeae0d5f97bbf437d4a338fd4128ad" Oct 30 18:56:00.722742 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:00.722656 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7496fa-jbqbv" event={"ID":"151d129d-222e-4c7b-91f1-828d24137732","Type":"ContainerDied","Data":"61f25af9af5c120e593a428bc0ac81ca58a2bf903bb4cda76046d024a5891fa8"} Oct 30 18:56:00.722742 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:00.722709 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61f25af9af5c120e593a428bc0ac81ca58a2bf903bb4cda76046d024a5891fa8" Oct 30 18:56:00.722742 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:00.722673 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7496fa-jbqbv" Oct 30 18:56:17.871470 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:17.871433 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv"] Oct 30 18:56:17.872112 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:17.871936 4531 cpu_manager.go:414] "containerMap: removing container" podUID="151d129d-222e-4c7b-91f1-828d24137732" containerName="revoke-keys" Oct 30 18:56:17.872112 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:17.871947 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151d129d-222e-4c7b-91f1-828d24137732" containerName="revoke-keys" Oct 30 18:56:17.872112 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:17.872030 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="151d129d-222e-4c7b-91f1-828d24137732" containerName="revoke-keys" Oct 30 18:56:17.877371 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:17.877338 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv" Oct 30 18:56:17.919217 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:17.919184 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f824x\" (UniqueName: \"kubernetes.io/projected/8b970059-ff75-4aba-b1c3-cda2f2534430-kube-api-access-f824x\") pod \"payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv\" (UID: \"8b970059-ff75-4aba-b1c3-cda2f2534430\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv" Oct 30 18:56:17.919397 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:17.919238 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b970059-ff75-4aba-b1c3-cda2f2534430-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv\" (UID: \"8b970059-ff75-4aba-b1c3-cda2f2534430\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv" Oct 30 18:56:17.934877 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:17.934835 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv"] Oct 30 18:56:18.017859 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:18.017821 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-10236a-dockercfg-zc89t\"" Oct 30 18:56:18.024503 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:18.018893 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-10236a\"" Oct 30 18:56:18.031163 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:18.031114 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f824x\" (UniqueName: \"kubernetes.io/projected/8b970059-ff75-4aba-b1c3-cda2f2534430-kube-api-access-f824x\") pod \"payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv\" (UID: \"8b970059-ff75-4aba-b1c3-cda2f2534430\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv" Oct 30 18:56:18.031903 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:18.031875 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b970059-ff75-4aba-b1c3-cda2f2534430-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv\" (UID: \"8b970059-ff75-4aba-b1c3-cda2f2534430\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv" Oct 30 18:56:18.062155 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:18.052209 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw"] Oct 30 18:56:18.062155 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:18.060567 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b970059-ff75-4aba-b1c3-cda2f2534430-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv\" (UID: \"8b970059-ff75-4aba-b1c3-cda2f2534430\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv" Oct 30 18:56:18.075883 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:18.075292 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw"] Oct 30 18:56:18.075883 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:18.075426 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw" Oct 30 18:56:18.093531 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:18.093504 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f824x\" (UniqueName: \"kubernetes.io/projected/8b970059-ff75-4aba-b1c3-cda2f2534430-kube-api-access-f824x\") pod \"payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv\" (UID: \"8b970059-ff75-4aba-b1c3-cda2f2534430\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv" Oct 30 18:56:18.146648 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:18.146558 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tnrs\" (UniqueName: \"kubernetes.io/projected/7d1e0b02-28ed-4d52-b315-663f7db2d945-kube-api-access-5tnrs\") pod \"payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw\" (UID: \"7d1e0b02-28ed-4d52-b315-663f7db2d945\") " pod="openshift-ingress/payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw" Oct 30 18:56:18.146850 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:18.146680 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d1e0b02-28ed-4d52-b315-663f7db2d945-plugins-config-volume\") pod \"payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw\" (UID: \"7d1e0b02-28ed-4d52-b315-663f7db2d945\") " pod="openshift-ingress/payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw" Oct 30 18:56:18.192023 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:18.191983 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv" Oct 30 18:56:18.248651 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:18.248615 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d1e0b02-28ed-4d52-b315-663f7db2d945-plugins-config-volume\") pod \"payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw\" (UID: \"7d1e0b02-28ed-4d52-b315-663f7db2d945\") " pod="openshift-ingress/payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw" Oct 30 18:56:18.248844 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:18.248760 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tnrs\" (UniqueName: \"kubernetes.io/projected/7d1e0b02-28ed-4d52-b315-663f7db2d945-kube-api-access-5tnrs\") pod \"payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw\" (UID: \"7d1e0b02-28ed-4d52-b315-663f7db2d945\") " pod="openshift-ingress/payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw" Oct 30 18:56:18.249606 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:18.249572 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d1e0b02-28ed-4d52-b315-663f7db2d945-plugins-config-volume\") pod \"payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw\" (UID: \"7d1e0b02-28ed-4d52-b315-663f7db2d945\") " pod="openshift-ingress/payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw" Oct 30 18:56:18.312279 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:18.312196 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tnrs\" (UniqueName: \"kubernetes.io/projected/7d1e0b02-28ed-4d52-b315-663f7db2d945-kube-api-access-5tnrs\") pod \"payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw\" (UID: \"7d1e0b02-28ed-4d52-b315-663f7db2d945\") " pod="openshift-ingress/payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw" Oct 30 18:56:18.395989 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:18.395952 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw" Oct 30 18:56:18.591761 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:18.591721 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv"] Oct 30 18:56:18.592872 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:56:18.592847 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b970059_ff75_4aba_b1c3_cda2f2534430.slice/crio-e950d00ebb44e0213ad4e6f3fa4456b127663fefb16fc5d6506842215ece9baa WatchSource:0}: Error finding container e950d00ebb44e0213ad4e6f3fa4456b127663fefb16fc5d6506842215ece9baa: Status 404 returned error can't find the container with id e950d00ebb44e0213ad4e6f3fa4456b127663fefb16fc5d6506842215ece9baa Oct 30 18:56:18.778741 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:18.778708 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw"] Oct 30 18:56:18.779167 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:56:18.779109 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1e0b02_28ed_4d52_b315_663f7db2d945.slice/crio-a4d0e7aa57fe5bd75ef4f3ae9dd3b857c01cc279863f3390b77dcd48bc932c31 WatchSource:0}: Error finding container a4d0e7aa57fe5bd75ef4f3ae9dd3b857c01cc279863f3390b77dcd48bc932c31: Status 404 returned error can't find the container with id a4d0e7aa57fe5bd75ef4f3ae9dd3b857c01cc279863f3390b77dcd48bc932c31 Oct 30 18:56:18.881781 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:18.881738 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv" event={"ID":"8b970059-ff75-4aba-b1c3-cda2f2534430","Type":"ContainerStarted","Data":"ee29d2b09aaab897ebe1e1e91d552f424536bf4f74f5c07d5aadccdd577b5b6a"} Oct 30 18:56:18.882317 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:18.881809 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv" event={"ID":"8b970059-ff75-4aba-b1c3-cda2f2534430","Type":"ContainerStarted","Data":"e950d00ebb44e0213ad4e6f3fa4456b127663fefb16fc5d6506842215ece9baa"} Oct 30 18:56:18.882317 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:18.881839 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv" Oct 30 18:56:18.883709 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:18.883674 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw" event={"ID":"7d1e0b02-28ed-4d52-b315-663f7db2d945","Type":"ContainerStarted","Data":"bcb8e893c138e4c4d42baf31ac94ee7adabc1cd1aba835f79db04142e5fb2e53"} Oct 30 18:56:18.883875 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:18.883737 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw" event={"ID":"7d1e0b02-28ed-4d52-b315-663f7db2d945","Type":"ContainerStarted","Data":"a4d0e7aa57fe5bd75ef4f3ae9dd3b857c01cc279863f3390b77dcd48bc932c31"} Oct 30 18:56:18.883875 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:18.883781 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw" Oct 30 18:56:19.000180 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:19.000108 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv" podStartSLOduration=2.000092934 podStartE2EDuration="2.000092934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:18.99755969 +0000 UTC m=+1339.024664646" watchObservedRunningTime="2026-10-30 18:56:19.000092934 +0000 UTC m=+1339.027197954" Oct 30 18:56:19.114400 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:19.114343 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw" podStartSLOduration=2.114327518 podStartE2EDuration="2.114327518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:19.11374268 +0000 UTC m=+1339.140847639" watchObservedRunningTime="2026-10-30 18:56:19.114327518 +0000 UTC m=+1339.141432476" Oct 30 18:56:21.264244 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:21.264214 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6972f60f-nnp8v"] Oct 30 18:56:21.268266 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:21.268245 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6972f60f-nnp8v" Oct 30 18:56:21.323386 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:21.323246 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:56:21.323386 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:21.323312 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:56:21.323701 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:21.323398 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:56:21.323701 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:21.323562 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:56:21.327837 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:21.327797 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6972f60f-nnp8v"] Oct 30 18:56:21.388403 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:21.388356 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85de0aae-f351-4133-b2cc-73767de64c96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6972f60f-nnp8v\" (UID: \"85de0aae-f351-4133-b2cc-73767de64c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6972f60f-nnp8v" Oct 30 18:56:21.388643 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:21.388622 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85de0aae-f351-4133-b2cc-73767de64c96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6972f60f-nnp8v\" (UID: \"85de0aae-f351-4133-b2cc-73767de64c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6972f60f-nnp8v" Oct 30 18:56:21.388774 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:21.388753 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s67z\" (UniqueName: \"kubernetes.io/projected/85de0aae-f351-4133-b2cc-73767de64c96-kube-api-access-2s67z\") pod \"maas-api-revoke-sub-6972f60f-nnp8v\" (UID: \"85de0aae-f351-4133-b2cc-73767de64c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6972f60f-nnp8v" Oct 30 18:56:21.490011 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:21.489977 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85de0aae-f351-4133-b2cc-73767de64c96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6972f60f-nnp8v\" (UID: \"85de0aae-f351-4133-b2cc-73767de64c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6972f60f-nnp8v" Oct 30 18:56:21.490199 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:21.490049 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s67z\" (UniqueName: \"kubernetes.io/projected/85de0aae-f351-4133-b2cc-73767de64c96-kube-api-access-2s67z\") pod \"maas-api-revoke-sub-6972f60f-nnp8v\" (UID: \"85de0aae-f351-4133-b2cc-73767de64c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6972f60f-nnp8v" Oct 30 18:56:21.490199 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:21.490085 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85de0aae-f351-4133-b2cc-73767de64c96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6972f60f-nnp8v\" (UID: \"85de0aae-f351-4133-b2cc-73767de64c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6972f60f-nnp8v" Oct 30 18:56:21.490812 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:21.490785 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85de0aae-f351-4133-b2cc-73767de64c96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6972f60f-nnp8v\" (UID: \"85de0aae-f351-4133-b2cc-73767de64c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6972f60f-nnp8v" Oct 30 18:56:21.492649 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:21.492628 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85de0aae-f351-4133-b2cc-73767de64c96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6972f60f-nnp8v\" (UID: \"85de0aae-f351-4133-b2cc-73767de64c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6972f60f-nnp8v" Oct 30 18:56:21.551242 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:21.551159 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s67z\" (UniqueName: \"kubernetes.io/projected/85de0aae-f351-4133-b2cc-73767de64c96-kube-api-access-2s67z\") pod \"maas-api-revoke-sub-6972f60f-nnp8v\" (UID: \"85de0aae-f351-4133-b2cc-73767de64c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6972f60f-nnp8v" Oct 30 18:56:21.583276 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:21.583241 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6972f60f-nnp8v" Oct 30 18:56:21.961220 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:21.961192 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6972f60f-nnp8v"] Oct 30 18:56:21.962147 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:56:21.962103 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85de0aae_f351_4133_b2cc_73767de64c96.slice/crio-1e1da62ec9da4d925be649ba95c0fe5b9e7e35e2555df3e1b3a291d0ae33c51f WatchSource:0}: Error finding container 1e1da62ec9da4d925be649ba95c0fe5b9e7e35e2555df3e1b3a291d0ae33c51f: Status 404 returned error can't find the container with id 1e1da62ec9da4d925be649ba95c0fe5b9e7e35e2555df3e1b3a291d0ae33c51f Oct 30 18:56:22.911642 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:22.911601 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6972f60f-nnp8v" event={"ID":"85de0aae-f351-4133-b2cc-73767de64c96","Type":"ContainerStarted","Data":"fdf761f8a30c040544d6ad66342286e8eadbb85a8cf815981f0a6a84e63c3486"} Oct 30 18:56:22.912201 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:22.911684 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6972f60f-nnp8v" event={"ID":"85de0aae-f351-4133-b2cc-73767de64c96","Type":"ContainerStarted","Data":"1e1da62ec9da4d925be649ba95c0fe5b9e7e35e2555df3e1b3a291d0ae33c51f"} Oct 30 18:56:23.030640 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:23.030581 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6972f60f-nnp8v" podStartSLOduration=2.030566852 podStartE2EDuration="2.030566852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:23.02701729 +0000 UTC m=+1343.054122249" watchObservedRunningTime="2026-10-30 18:56:23.030566852 +0000 UTC m=+1343.057671810" Oct 30 18:56:23.921496 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:23.921455 4531 generic.go:356] "Generic (PLEG): container finished" podID="85de0aae-f351-4133-b2cc-73767de64c96" containerID="fdf761f8a30c040544d6ad66342286e8eadbb85a8cf815981f0a6a84e63c3486" exitCode=0 Oct 30 18:56:23.921911 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:23.921536 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6972f60f-nnp8v" event={"ID":"85de0aae-f351-4133-b2cc-73767de64c96","Type":"ContainerDied","Data":"fdf761f8a30c040544d6ad66342286e8eadbb85a8cf815981f0a6a84e63c3486"} Oct 30 18:56:25.092299 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.092266 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6972f60f-nnp8v" Oct 30 18:56:25.240047 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.239935 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85de0aae-f351-4133-b2cc-73767de64c96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85de0aae-f351-4133-b2cc-73767de64c96-maas-api-xks-ca\") pod \"85de0aae-f351-4133-b2cc-73767de64c96\" (UID: \"85de0aae-f351-4133-b2cc-73767de64c96\") " Oct 30 18:56:25.240270 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.240106 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85de0aae-f351-4133-b2cc-73767de64c96-kube-api-access-2s67z\" (UniqueName: \"kubernetes.io/projected/85de0aae-f351-4133-b2cc-73767de64c96-kube-api-access-2s67z\") pod \"85de0aae-f351-4133-b2cc-73767de64c96\" (UID: \"85de0aae-f351-4133-b2cc-73767de64c96\") " Oct 30 18:56:25.240270 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.240199 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85de0aae-f351-4133-b2cc-73767de64c96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85de0aae-f351-4133-b2cc-73767de64c96-maas-api-openshift-ca\") pod \"85de0aae-f351-4133-b2cc-73767de64c96\" (UID: \"85de0aae-f351-4133-b2cc-73767de64c96\") " Oct 30 18:56:25.240626 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.240584 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85de0aae-f351-4133-b2cc-73767de64c96-maas-api-openshift-ca" pod "85de0aae-f351-4133-b2cc-73767de64c96" (UID: "85de0aae-f351-4133-b2cc-73767de64c96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:56:25.240851 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.240832 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85de0aae-f351-4133-b2cc-73767de64c96-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:56:25.242517 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.242484 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85de0aae-f351-4133-b2cc-73767de64c96-maas-api-xks-ca" pod "85de0aae-f351-4133-b2cc-73767de64c96" (UID: "85de0aae-f351-4133-b2cc-73767de64c96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:25.242977 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.242958 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85de0aae-f351-4133-b2cc-73767de64c96-kube-api-access-2s67z" pod "85de0aae-f351-4133-b2cc-73767de64c96" (UID: "85de0aae-f351-4133-b2cc-73767de64c96"). InnerVolumeSpecName "kube-api-access-2s67z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:56:25.341643 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.341604 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85de0aae-f351-4133-b2cc-73767de64c96-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:56:25.341643 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.341636 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s67z\" (UniqueName: \"kubernetes.io/projected/85de0aae-f351-4133-b2cc-73767de64c96-kube-api-access-2s67z\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:56:25.632419 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.632366 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88"] Oct 30 18:56:25.633327 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.633306 4531 cpu_manager.go:414] "containerMap: removing container" podUID="85de0aae-f351-4133-b2cc-73767de64c96" containerName="revoke-keys" Oct 30 18:56:25.633467 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.633455 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85de0aae-f351-4133-b2cc-73767de64c96" containerName="revoke-keys" Oct 30 18:56:25.633708 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.633697 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="85de0aae-f351-4133-b2cc-73767de64c96" containerName="revoke-keys" Oct 30 18:56:25.653800 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.653768 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.698057 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.698019 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88"] Oct 30 18:56:25.754751 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.754709 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-credential-socket\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.754938 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.754857 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-token\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.754938 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.754917 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88fc33e1-96f8-4682-aedf-ef669f85138c-istiod-ca-cert\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.755040 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.754994 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-envoy\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.755076 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.755035 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-data\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.755160 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.755082 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-podinfo\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.755426 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.755400 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-workload-socket\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.755471 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.755441 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ft4x\" (UniqueName: \"kubernetes.io/projected/88fc33e1-96f8-4682-aedf-ef669f85138c-kube-api-access-9ft4x\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.755541 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.755520 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-workload-certs\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.857060 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.857015 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-workload-socket\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.857060 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.857059 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ft4x\" (UniqueName: \"kubernetes.io/projected/88fc33e1-96f8-4682-aedf-ef669f85138c-kube-api-access-9ft4x\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.857379 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.857083 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-workload-certs\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.857379 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.857169 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-credential-socket\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.857379 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.857234 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-token\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.857379 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.857256 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88fc33e1-96f8-4682-aedf-ef669f85138c-istiod-ca-cert\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.857379 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.857290 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-envoy\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.857379 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.857323 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-data\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.857729 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.857416 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-podinfo\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.857729 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.857553 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-workload-socket\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.857729 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.857661 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-workload-certs\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.857729 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.857691 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-credential-socket\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.857912 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.857742 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-data\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.858165 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.858122 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88fc33e1-96f8-4682-aedf-ef669f85138c-istiod-ca-cert\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.859958 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.859935 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-envoy\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.860077 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.860010 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-podinfo\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.918442 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.918364 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-token\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.918442 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.918384 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ft4x\" (UniqueName: \"kubernetes.io/projected/88fc33e1-96f8-4682-aedf-ef669f85138c-kube-api-access-9ft4x\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:25.939978 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.939929 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6972f60f-nnp8v" event={"ID":"85de0aae-f351-4133-b2cc-73767de64c96","Type":"ContainerDied","Data":"1e1da62ec9da4d925be649ba95c0fe5b9e7e35e2555df3e1b3a291d0ae33c51f"} Oct 30 18:56:25.940202 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.940039 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e1da62ec9da4d925be649ba95c0fe5b9e7e35e2555df3e1b3a291d0ae33c51f" Oct 30 18:56:25.940202 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.940157 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6972f60f-nnp8v" Oct 30 18:56:25.970650 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:25.970610 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:26.399671 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:56:26.399634 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:56:26.406405 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:26.406375 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88"] Oct 30 18:56:26.406560 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:56:26.406527 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88fc33e1_96f8_4682_aedf_ef669f85138c.slice/crio-812c5e327a567bb9835d87334d24e13b2255390a91ce60d90873e213447bf29b WatchSource:0}: Error finding container 812c5e327a567bb9835d87334d24e13b2255390a91ce60d90873e213447bf29b: Status 404 returned error can't find the container with id 812c5e327a567bb9835d87334d24e13b2255390a91ce60d90873e213447bf29b Oct 30 18:56:26.409100 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:26.409066 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:56:26.409201 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:26.409172 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:56:26.409255 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:26.409204 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:56:26.948324 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:26.948285 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" event={"ID":"88fc33e1-96f8-4682-aedf-ef669f85138c","Type":"ContainerStarted","Data":"940cbb9dabd7da95639c197d9454b7036ce152514e07bfd83f9241f66485b38b"} Oct 30 18:56:26.948550 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:26.948346 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" event={"ID":"88fc33e1-96f8-4682-aedf-ef669f85138c","Type":"ContainerStarted","Data":"812c5e327a567bb9835d87334d24e13b2255390a91ce60d90873e213447bf29b"} Oct 30 18:56:26.971762 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:26.971722 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:27.973403 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:27.973363 4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 18:56:27.973898 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:27.973437 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" podUID="88fc33e1-96f8-4682-aedf-ef669f85138c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 18:56:28.971811 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:28.971776 4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 18:56:28.972001 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:28.971843 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" podUID="88fc33e1-96f8-4682-aedf-ef669f85138c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 18:56:29.894273 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:29.894241 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv" Oct 30 18:56:29.894695 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:29.894327 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw" Oct 30 18:56:29.971913 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:29.971878 4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 18:56:29.972102 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:29.971936 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" podUID="88fc33e1-96f8-4682-aedf-ef669f85138c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 18:56:30.012974 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.012914 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" podStartSLOduration=5.012892551 podStartE2EDuration="5.012892551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:27.070609697 +0000 UTC m=+1347.097714656" watchObservedRunningTime="2026-10-30 18:56:30.012892551 +0000 UTC m=+1350.039997506" Oct 30 18:56:30.641655 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.641607 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx"] Oct 30 18:56:30.647145 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.647105 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" Oct 30 18:56:30.702267 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.702235 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx"] Oct 30 18:56:30.741884 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.741847 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-dshm\") pod \"e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx\" (UID: \"dab47e85-278b-4d4f-94b2-8f46ec744d69\") " pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" Oct 30 18:56:30.742205 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.742183 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-model-cache\") pod \"e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx\" (UID: \"dab47e85-278b-4d4f-94b2-8f46ec744d69\") " pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" Oct 30 18:56:30.742307 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.742291 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dab47e85-278b-4d4f-94b2-8f46ec744d69-tls-certs\") pod \"e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx\" (UID: \"dab47e85-278b-4d4f-94b2-8f46ec744d69\") " pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" Oct 30 18:56:30.742491 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.742471 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-tmp-dir\") pod \"e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx\" (UID: \"dab47e85-278b-4d4f-94b2-8f46ec744d69\") " pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" Oct 30 18:56:30.742639 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.742611 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmz97\" (UniqueName: \"kubernetes.io/projected/dab47e85-278b-4d4f-94b2-8f46ec744d69-kube-api-access-wmz97\") pod \"e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx\" (UID: \"dab47e85-278b-4d4f-94b2-8f46ec744d69\") " pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" Oct 30 18:56:30.742721 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.742679 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-home\") pod \"e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx\" (UID: \"dab47e85-278b-4d4f-94b2-8f46ec744d69\") " pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" Oct 30 18:56:30.747304 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.747273 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-66baf45c\"/\"openshift-service-ca.crt\"" Oct 30 18:56:30.747481 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.747378 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-66baf45c\"/\"kube-root-ca.crt\"" Oct 30 18:56:30.747553 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.747525 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-66baf45c\"/\"default-dockercfg-qkkrv\"" Oct 30 18:56:30.747553 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.747528 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-66baf45c\"/\"e2e-lifecycle-model-66baf45c-kserve-self-signed-certs\"" Oct 30 18:56:30.853154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.845957 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-tmp-dir\") pod \"e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx\" (UID: \"dab47e85-278b-4d4f-94b2-8f46ec744d69\") " pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" Oct 30 18:56:30.853154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.846016 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmz97\" (UniqueName: \"kubernetes.io/projected/dab47e85-278b-4d4f-94b2-8f46ec744d69-kube-api-access-wmz97\") pod \"e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx\" (UID: \"dab47e85-278b-4d4f-94b2-8f46ec744d69\") " pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" Oct 30 18:56:30.853154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.846071 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-home\") pod \"e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx\" (UID: \"dab47e85-278b-4d4f-94b2-8f46ec744d69\") " pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" Oct 30 18:56:30.853154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.846176 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-dshm\") pod \"e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx\" (UID: \"dab47e85-278b-4d4f-94b2-8f46ec744d69\") " pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" Oct 30 18:56:30.853154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.846239 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-model-cache\") pod \"e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx\" (UID: \"dab47e85-278b-4d4f-94b2-8f46ec744d69\") " pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" Oct 30 18:56:30.853154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.846284 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dab47e85-278b-4d4f-94b2-8f46ec744d69-tls-certs\") pod \"e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx\" (UID: \"dab47e85-278b-4d4f-94b2-8f46ec744d69\") " pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" Oct 30 18:56:30.853154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.846707 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-home\") pod \"e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx\" (UID: \"dab47e85-278b-4d4f-94b2-8f46ec744d69\") " pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" Oct 30 18:56:30.853154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.846966 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-tmp-dir\") pod \"e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx\" (UID: \"dab47e85-278b-4d4f-94b2-8f46ec744d69\") " pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" Oct 30 18:56:30.853154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.848270 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-model-cache\") pod \"e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx\" (UID: \"dab47e85-278b-4d4f-94b2-8f46ec744d69\") " pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" Oct 30 18:56:30.859212 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.858397 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dab47e85-278b-4d4f-94b2-8f46ec744d69-tls-certs\") pod \"e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx\" (UID: \"dab47e85-278b-4d4f-94b2-8f46ec744d69\") " pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" Oct 30 18:56:30.865186 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.861453 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-dshm\") pod \"e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx\" (UID: \"dab47e85-278b-4d4f-94b2-8f46ec744d69\") " pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" Oct 30 18:56:30.909442 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.909338 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmz97\" (UniqueName: \"kubernetes.io/projected/dab47e85-278b-4d4f-94b2-8f46ec744d69-kube-api-access-wmz97\") pod \"e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx\" (UID: \"dab47e85-278b-4d4f-94b2-8f46ec744d69\") " pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" Oct 30 18:56:30.961113 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.960982 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" Oct 30 18:56:30.971469 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.971430 4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 18:56:30.971659 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:30.971512 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" podUID="88fc33e1-96f8-4682-aedf-ef669f85138c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 18:56:31.395165 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:31.395118 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx"] Oct 30 18:56:31.396147 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:56:31.396099 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddab47e85_278b_4d4f_94b2_8f46ec744d69.slice/crio-94a0f09b441caacea8e69b4962b0eaafe399033af197127ede4bb94510217dd1 WatchSource:0}: Error finding container 94a0f09b441caacea8e69b4962b0eaafe399033af197127ede4bb94510217dd1: Status 404 returned error can't find the container with id 94a0f09b441caacea8e69b4962b0eaafe399033af197127ede4bb94510217dd1 Oct 30 18:56:31.971867 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:31.971832 4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 18:56:31.972378 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:31.971900 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" podUID="88fc33e1-96f8-4682-aedf-ef669f85138c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 18:56:31.984168 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:31.984111 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" event={"ID":"dab47e85-278b-4d4f-94b2-8f46ec744d69","Type":"ContainerStarted","Data":"8328d8383b5c5d699465f3924a5e6c2a8c764a0347d102ded9a2fc8e5d069ad1"} Oct 30 18:56:31.984168 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:31.984180 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" event={"ID":"dab47e85-278b-4d4f-94b2-8f46ec744d69","Type":"ContainerStarted","Data":"94a0f09b441caacea8e69b4962b0eaafe399033af197127ede4bb94510217dd1"} Oct 30 18:56:32.104647 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:32.104592 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" podStartSLOduration=2.104576751 podStartE2EDuration="2.104576751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:32.103588233 +0000 UTC m=+1352.130693192" watchObservedRunningTime="2026-10-30 18:56:32.104576751 +0000 UTC m=+1352.131681710" Oct 30 18:56:32.961666 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:32.961625 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" Oct 30 18:56:32.971197 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:32.971104 4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 18:56:32.971351 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:32.971241 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" podUID="88fc33e1-96f8-4682-aedf-ef669f85138c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 18:56:33.999294 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:33.999261 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:34.000288 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:34.000268 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:34.001898 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:34.001875 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 18:56:40.260937 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:40.260904 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4302db-lb7rb"] Oct 30 18:56:40.264890 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:40.264870 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4302db-lb7rb" Oct 30 18:56:40.368802 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:40.368769 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:56:40.368802 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:40.368787 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:56:40.368802 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:40.368801 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:56:40.369121 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:40.368870 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:56:40.371460 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:40.371434 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b18c67b-7a80-4174-a16f-18219c650cb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d4302db-lb7rb\" (UID: \"8b18c67b-7a80-4174-a16f-18219c650cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4302db-lb7rb" Oct 30 18:56:40.371608 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:40.371583 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zvfb\" (UniqueName: \"kubernetes.io/projected/8b18c67b-7a80-4174-a16f-18219c650cb4-kube-api-access-6zvfb\") pod \"maas-api-revoke-sub-8d4302db-lb7rb\" (UID: \"8b18c67b-7a80-4174-a16f-18219c650cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4302db-lb7rb" Oct 30 18:56:40.371671 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:40.371651 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b18c67b-7a80-4174-a16f-18219c650cb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d4302db-lb7rb\" (UID: \"8b18c67b-7a80-4174-a16f-18219c650cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4302db-lb7rb" Oct 30 18:56:40.372915 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:40.372889 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4302db-lb7rb"] Oct 30 18:56:40.472550 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:40.472511 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b18c67b-7a80-4174-a16f-18219c650cb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d4302db-lb7rb\" (UID: \"8b18c67b-7a80-4174-a16f-18219c650cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4302db-lb7rb" Oct 30 18:56:40.472707 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:40.472645 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zvfb\" (UniqueName: \"kubernetes.io/projected/8b18c67b-7a80-4174-a16f-18219c650cb4-kube-api-access-6zvfb\") pod \"maas-api-revoke-sub-8d4302db-lb7rb\" (UID: \"8b18c67b-7a80-4174-a16f-18219c650cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4302db-lb7rb" Oct 30 18:56:40.472707 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:40.472681 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b18c67b-7a80-4174-a16f-18219c650cb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d4302db-lb7rb\" (UID: \"8b18c67b-7a80-4174-a16f-18219c650cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4302db-lb7rb" Oct 30 18:56:40.473433 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:40.473414 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b18c67b-7a80-4174-a16f-18219c650cb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d4302db-lb7rb\" (UID: \"8b18c67b-7a80-4174-a16f-18219c650cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4302db-lb7rb" Oct 30 18:56:40.475190 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:40.475165 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b18c67b-7a80-4174-a16f-18219c650cb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d4302db-lb7rb\" (UID: \"8b18c67b-7a80-4174-a16f-18219c650cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4302db-lb7rb" Oct 30 18:56:40.537051 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:40.536983 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zvfb\" (UniqueName: \"kubernetes.io/projected/8b18c67b-7a80-4174-a16f-18219c650cb4-kube-api-access-6zvfb\") pod \"maas-api-revoke-sub-8d4302db-lb7rb\" (UID: \"8b18c67b-7a80-4174-a16f-18219c650cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4302db-lb7rb" Oct 30 18:56:40.579017 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:40.578981 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4302db-lb7rb" Oct 30 18:56:40.958202 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:40.958173 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4302db-lb7rb"] Oct 30 18:56:40.958752 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:56:40.958730 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b18c67b_7a80_4174_a16f_18219c650cb4.slice/crio-3a1ec8409be6546ceee7fad440cbcd2170d84136065a28e222895515f916c9f8 WatchSource:0}: Error finding container 3a1ec8409be6546ceee7fad440cbcd2170d84136065a28e222895515f916c9f8: Status 404 returned error can't find the container with id 3a1ec8409be6546ceee7fad440cbcd2170d84136065a28e222895515f916c9f8 Oct 30 18:56:40.961280 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:40.961255 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" Oct 30 18:56:40.976952 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:40.976930 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" Oct 30 18:56:41.047271 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:41.047235 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4302db-lb7rb" event={"ID":"8b18c67b-7a80-4174-a16f-18219c650cb4","Type":"ContainerStarted","Data":"3a1ec8409be6546ceee7fad440cbcd2170d84136065a28e222895515f916c9f8"} Oct 30 18:56:41.058957 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:41.058928 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" Oct 30 18:56:42.056093 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:42.056059 4531 generic.go:356] "Generic (PLEG): container finished" podID="8b18c67b-7a80-4174-a16f-18219c650cb4" containerID="3134c6f1984b1ffe0bbf01174bf1bf9d8d923cf76d67b78e33803f29d9c5b4fd" exitCode=0 Oct 30 18:56:42.056606 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:42.056164 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4302db-lb7rb" event={"ID":"8b18c67b-7a80-4174-a16f-18219c650cb4","Type":"ContainerDied","Data":"3134c6f1984b1ffe0bbf01174bf1bf9d8d923cf76d67b78e33803f29d9c5b4fd"} Oct 30 18:56:43.212757 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:43.212734 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4302db-lb7rb" Oct 30 18:56:43.306754 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:43.306718 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b18c67b-7a80-4174-a16f-18219c650cb4-kube-api-access-6zvfb\" (UniqueName: \"kubernetes.io/projected/8b18c67b-7a80-4174-a16f-18219c650cb4-kube-api-access-6zvfb\") pod \"8b18c67b-7a80-4174-a16f-18219c650cb4\" (UID: \"8b18c67b-7a80-4174-a16f-18219c650cb4\") " Oct 30 18:56:43.306908 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:43.306837 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b18c67b-7a80-4174-a16f-18219c650cb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b18c67b-7a80-4174-a16f-18219c650cb4-maas-api-openshift-ca\") pod \"8b18c67b-7a80-4174-a16f-18219c650cb4\" (UID: \"8b18c67b-7a80-4174-a16f-18219c650cb4\") " Oct 30 18:56:43.306966 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:43.306951 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b18c67b-7a80-4174-a16f-18219c650cb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b18c67b-7a80-4174-a16f-18219c650cb4-maas-api-xks-ca\") pod \"8b18c67b-7a80-4174-a16f-18219c650cb4\" (UID: \"8b18c67b-7a80-4174-a16f-18219c650cb4\") " Oct 30 18:56:43.307251 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:43.307229 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b18c67b-7a80-4174-a16f-18219c650cb4-maas-api-openshift-ca" pod "8b18c67b-7a80-4174-a16f-18219c650cb4" (UID: "8b18c67b-7a80-4174-a16f-18219c650cb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:56:43.307475 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:43.307431 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b18c67b-7a80-4174-a16f-18219c650cb4-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:56:43.309040 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:43.309015 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b18c67b-7a80-4174-a16f-18219c650cb4-kube-api-access-6zvfb" pod "8b18c67b-7a80-4174-a16f-18219c650cb4" (UID: "8b18c67b-7a80-4174-a16f-18219c650cb4"). InnerVolumeSpecName "kube-api-access-6zvfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:56:43.309158 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:43.309086 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b18c67b-7a80-4174-a16f-18219c650cb4-maas-api-xks-ca" pod "8b18c67b-7a80-4174-a16f-18219c650cb4" (UID: "8b18c67b-7a80-4174-a16f-18219c650cb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:43.408988 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:43.408945 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b18c67b-7a80-4174-a16f-18219c650cb4-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:56:43.408988 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:43.408981 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zvfb\" (UniqueName: \"kubernetes.io/projected/8b18c67b-7a80-4174-a16f-18219c650cb4-kube-api-access-6zvfb\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:56:44.070770 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:44.070737 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4302db-lb7rb" Oct 30 18:56:44.070770 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:44.070756 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4302db-lb7rb" event={"ID":"8b18c67b-7a80-4174-a16f-18219c650cb4","Type":"ContainerDied","Data":"3a1ec8409be6546ceee7fad440cbcd2170d84136065a28e222895515f916c9f8"} Oct 30 18:56:44.070988 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:44.070802 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a1ec8409be6546ceee7fad440cbcd2170d84136065a28e222895515f916c9f8" Oct 30 18:56:51.005103 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:51.005057 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf555aa7-p47r8"] Oct 30 18:56:51.005673 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:51.005659 4531 cpu_manager.go:414] "containerMap: removing container" podUID="8b18c67b-7a80-4174-a16f-18219c650cb4" containerName="revoke-keys" Oct 30 18:56:51.005716 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:51.005673 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b18c67b-7a80-4174-a16f-18219c650cb4" containerName="revoke-keys" Oct 30 18:56:51.005773 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:51.005762 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b18c67b-7a80-4174-a16f-18219c650cb4" containerName="revoke-keys" Oct 30 18:56:51.009529 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:51.009508 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf555aa7-p47r8" Oct 30 18:56:51.104552 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:51.104512 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87f754d5-f500-47c8-9026-07737a66b6ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf555aa7-p47r8\" (UID: \"87f754d5-f500-47c8-9026-07737a66b6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf555aa7-p47r8" Oct 30 18:56:51.104762 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:51.104559 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxsk9\" (UniqueName: \"kubernetes.io/projected/87f754d5-f500-47c8-9026-07737a66b6ea-kube-api-access-bxsk9\") pod \"maas-api-revoke-sub-bf555aa7-p47r8\" (UID: \"87f754d5-f500-47c8-9026-07737a66b6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf555aa7-p47r8" Oct 30 18:56:51.104762 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:51.104715 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87f754d5-f500-47c8-9026-07737a66b6ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf555aa7-p47r8\" (UID: \"87f754d5-f500-47c8-9026-07737a66b6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf555aa7-p47r8" Oct 30 18:56:51.112443 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:51.112415 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:56:51.112443 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:51.112413 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:56:51.112663 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:51.112475 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:56:51.112663 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:51.112496 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:56:51.116527 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:51.116500 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf555aa7-p47r8"] Oct 30 18:56:51.205635 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:51.205598 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87f754d5-f500-47c8-9026-07737a66b6ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf555aa7-p47r8\" (UID: \"87f754d5-f500-47c8-9026-07737a66b6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf555aa7-p47r8" Oct 30 18:56:51.205635 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:51.205637 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxsk9\" (UniqueName: \"kubernetes.io/projected/87f754d5-f500-47c8-9026-07737a66b6ea-kube-api-access-bxsk9\") pod \"maas-api-revoke-sub-bf555aa7-p47r8\" (UID: \"87f754d5-f500-47c8-9026-07737a66b6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf555aa7-p47r8" Oct 30 18:56:51.205846 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:51.205721 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87f754d5-f500-47c8-9026-07737a66b6ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf555aa7-p47r8\" (UID: \"87f754d5-f500-47c8-9026-07737a66b6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf555aa7-p47r8" Oct 30 18:56:51.206495 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:51.206474 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87f754d5-f500-47c8-9026-07737a66b6ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf555aa7-p47r8\" (UID: \"87f754d5-f500-47c8-9026-07737a66b6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf555aa7-p47r8" Oct 30 18:56:51.208349 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:51.208322 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87f754d5-f500-47c8-9026-07737a66b6ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf555aa7-p47r8\" (UID: \"87f754d5-f500-47c8-9026-07737a66b6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf555aa7-p47r8" Oct 30 18:56:51.268331 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:51.268254 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxsk9\" (UniqueName: \"kubernetes.io/projected/87f754d5-f500-47c8-9026-07737a66b6ea-kube-api-access-bxsk9\") pod \"maas-api-revoke-sub-bf555aa7-p47r8\" (UID: \"87f754d5-f500-47c8-9026-07737a66b6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf555aa7-p47r8" Oct 30 18:56:51.323792 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:51.323745 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf555aa7-p47r8" Oct 30 18:56:51.709557 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:56:51.709515 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87f754d5_f500_47c8_9026_07737a66b6ea.slice/crio-43c85b8e5225143c7dcaa3f68b2b1786e766aa836e518c04a2960b715bfa6cd8 WatchSource:0}: Error finding container 43c85b8e5225143c7dcaa3f68b2b1786e766aa836e518c04a2960b715bfa6cd8: Status 404 returned error can't find the container with id 43c85b8e5225143c7dcaa3f68b2b1786e766aa836e518c04a2960b715bfa6cd8 Oct 30 18:56:51.709750 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:51.709722 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf555aa7-p47r8"] Oct 30 18:56:52.214020 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:52.213982 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf555aa7-p47r8" event={"ID":"87f754d5-f500-47c8-9026-07737a66b6ea","Type":"ContainerStarted","Data":"468794aa7e638429af14aee6c342e56e2de19d8cdcf02ea675e97b6a5e06a1a6"} Oct 30 18:56:52.214535 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:52.214033 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf555aa7-p47r8" event={"ID":"87f754d5-f500-47c8-9026-07737a66b6ea","Type":"ContainerStarted","Data":"43c85b8e5225143c7dcaa3f68b2b1786e766aa836e518c04a2960b715bfa6cd8"} Oct 30 18:56:52.334817 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:52.334710 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf555aa7-p47r8" podStartSLOduration=2.334693496 podStartE2EDuration="2.334693496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:52.331178745 +0000 UTC m=+1372.358283705" watchObservedRunningTime="2026-10-30 18:56:52.334693496 +0000 UTC m=+1372.361798497" Oct 30 18:56:53.222558 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:53.222526 4531 generic.go:356] "Generic (PLEG): container finished" podID="87f754d5-f500-47c8-9026-07737a66b6ea" containerID="468794aa7e638429af14aee6c342e56e2de19d8cdcf02ea675e97b6a5e06a1a6" exitCode=0 Oct 30 18:56:53.222976 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:53.222612 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf555aa7-p47r8" event={"ID":"87f754d5-f500-47c8-9026-07737a66b6ea","Type":"ContainerDied","Data":"468794aa7e638429af14aee6c342e56e2de19d8cdcf02ea675e97b6a5e06a1a6"} Oct 30 18:56:54.392630 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:54.392606 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf555aa7-p47r8" Oct 30 18:56:54.445506 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:54.445475 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87f754d5-f500-47c8-9026-07737a66b6ea-kube-api-access-bxsk9\" (UniqueName: \"kubernetes.io/projected/87f754d5-f500-47c8-9026-07737a66b6ea-kube-api-access-bxsk9\") pod \"87f754d5-f500-47c8-9026-07737a66b6ea\" (UID: \"87f754d5-f500-47c8-9026-07737a66b6ea\") " Oct 30 18:56:54.445693 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:54.445538 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87f754d5-f500-47c8-9026-07737a66b6ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87f754d5-f500-47c8-9026-07737a66b6ea-maas-api-openshift-ca\") pod \"87f754d5-f500-47c8-9026-07737a66b6ea\" (UID: \"87f754d5-f500-47c8-9026-07737a66b6ea\") " Oct 30 18:56:54.445693 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:54.445601 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87f754d5-f500-47c8-9026-07737a66b6ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87f754d5-f500-47c8-9026-07737a66b6ea-maas-api-xks-ca\") pod \"87f754d5-f500-47c8-9026-07737a66b6ea\" (UID: \"87f754d5-f500-47c8-9026-07737a66b6ea\") " Oct 30 18:56:54.445949 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:54.445920 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87f754d5-f500-47c8-9026-07737a66b6ea-maas-api-openshift-ca" pod "87f754d5-f500-47c8-9026-07737a66b6ea" (UID: "87f754d5-f500-47c8-9026-07737a66b6ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:56:54.447877 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:54.447846 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87f754d5-f500-47c8-9026-07737a66b6ea-maas-api-xks-ca" pod "87f754d5-f500-47c8-9026-07737a66b6ea" (UID: "87f754d5-f500-47c8-9026-07737a66b6ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:54.447969 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:54.447914 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87f754d5-f500-47c8-9026-07737a66b6ea-kube-api-access-bxsk9" pod "87f754d5-f500-47c8-9026-07737a66b6ea" (UID: "87f754d5-f500-47c8-9026-07737a66b6ea"). InnerVolumeSpecName "kube-api-access-bxsk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:56:54.547666 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:54.547621 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxsk9\" (UniqueName: \"kubernetes.io/projected/87f754d5-f500-47c8-9026-07737a66b6ea-kube-api-access-bxsk9\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:56:54.547666 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:54.547653 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87f754d5-f500-47c8-9026-07737a66b6ea-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:56:54.547666 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:54.547664 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87f754d5-f500-47c8-9026-07737a66b6ea-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:56:55.238115 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:55.238087 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf555aa7-p47r8" Oct 30 18:56:55.238353 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:55.238146 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf555aa7-p47r8" event={"ID":"87f754d5-f500-47c8-9026-07737a66b6ea","Type":"ContainerDied","Data":"43c85b8e5225143c7dcaa3f68b2b1786e766aa836e518c04a2960b715bfa6cd8"} Oct 30 18:56:55.238353 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:56:55.238193 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43c85b8e5225143c7dcaa3f68b2b1786e766aa836e518c04a2960b715bfa6cd8" Oct 30 18:56:56.398757 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:56:56.398721 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:57:00.426883 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:00.426851 4531 scope.go:122] "RemoveContainer" containerID="59e71809aa2e928b3cfffbc89b62198e3313400d9cf9e9e5cf166455508efc08" Oct 30 18:57:05.053101 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:05.053063 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3477ba14-xjrlt"] Oct 30 18:57:05.103558 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:05.103516 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3477ba14-xjrlt"] Oct 30 18:57:06.099840 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:06.099806 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7913969-bc03-4c2a-8d9a-986941af9924" path="/var/lib/kubelet/pods/c7913969-bc03-4c2a-8d9a-986941af9924/volumes" Oct 30 18:57:11.006348 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.006262 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb"] Oct 30 18:57:11.006793 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.006778 4531 cpu_manager.go:414] "containerMap: removing container" podUID="87f754d5-f500-47c8-9026-07737a66b6ea" containerName="revoke-keys" Oct 30 18:57:11.006849 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.006794 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87f754d5-f500-47c8-9026-07737a66b6ea" containerName="revoke-keys" Oct 30 18:57:11.006900 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.006890 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="87f754d5-f500-47c8-9026-07737a66b6ea" containerName="revoke-keys" Oct 30 18:57:11.012091 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.012069 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.159940 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.159899 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-credential-socket\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.160115 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.159948 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-workload-socket\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.160115 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.160013 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhz8k\" (UniqueName: \"kubernetes.io/projected/ed1aa9fc-8605-4798-8b39-b75f2dde0337-kube-api-access-qhz8k\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.160115 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.160078 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-token\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.160280 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.160173 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-data\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.160280 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.160210 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-envoy\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.160354 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.160333 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-workload-certs\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.160391 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.160385 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-podinfo\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.160517 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.160502 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istiod-ca-cert\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.224919 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.224880 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb"] Oct 30 18:57:11.263030 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.262939 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-token\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.263030 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.262983 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-data\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.263030 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.263022 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-envoy\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.263396 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.263076 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-workload-certs\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.263598 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.263568 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-podinfo\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.263674 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.263617 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istiod-ca-cert\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.263674 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.263582 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-data\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.263674 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.263653 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-workload-certs\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.263997 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.263825 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-credential-socket\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.263997 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.263856 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-workload-socket\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.263997 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.263886 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhz8k\" (UniqueName: \"kubernetes.io/projected/ed1aa9fc-8605-4798-8b39-b75f2dde0337-kube-api-access-qhz8k\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.264457 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.264432 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istiod-ca-cert\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.264638 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.264475 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-credential-socket\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.264756 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.264520 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-workload-socket\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.265825 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.265796 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-envoy\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.266557 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.266532 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-podinfo\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.324437 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.324403 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-token\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.324593 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.324408 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhz8k\" (UniqueName: \"kubernetes.io/projected/ed1aa9fc-8605-4798-8b39-b75f2dde0337-kube-api-access-qhz8k\") pod \"e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.326339 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.326312 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:11.760244 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.760216 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb"] Oct 30 18:57:11.760739 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:57:11.760714 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded1aa9fc_8605_4798_8b39_b75f2dde0337.slice/crio-4881d795d1ca056b213284a1178d4583d4811b0d11c08afd1e024c18b83a5d12 WatchSource:0}: Error finding container 4881d795d1ca056b213284a1178d4583d4811b0d11c08afd1e024c18b83a5d12: Status 404 returned error can't find the container with id 4881d795d1ca056b213284a1178d4583d4811b0d11c08afd1e024c18b83a5d12 Oct 30 18:57:11.763279 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.763244 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:57:11.763352 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.763317 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:57:11.763390 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:11.763351 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:57:12.356618 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:12.356581 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" event={"ID":"ed1aa9fc-8605-4798-8b39-b75f2dde0337","Type":"ContainerStarted","Data":"4c43f6c64c0fc366396e836fe1c8cddbc85d7fef47b28b061a7fc186bb9fe5f5"} Oct 30 18:57:12.357047 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:12.356631 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" event={"ID":"ed1aa9fc-8605-4798-8b39-b75f2dde0337","Type":"ContainerStarted","Data":"4881d795d1ca056b213284a1178d4583d4811b0d11c08afd1e024c18b83a5d12"} Oct 30 18:57:12.485809 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:12.485742 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" podStartSLOduration=2.485721015 podStartE2EDuration="2.485721015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:12.481411322 +0000 UTC m=+1392.508516281" watchObservedRunningTime="2026-10-30 18:57:12.485721015 +0000 UTC m=+1392.512825976" Oct 30 18:57:13.087139 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:13.087086 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj"] Oct 30 18:57:13.091553 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:13.091529 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj" Oct 30 18:57:13.160493 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:13.160462 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-85ab6a-dockercfg-txv5s\"" Oct 30 18:57:13.160819 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:13.160804 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-85ab6a\"" Oct 30 18:57:13.168152 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:13.164639 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj"] Oct 30 18:57:13.213537 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:13.213498 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8zt8\" (UniqueName: \"kubernetes.io/projected/9c96011c-d91f-4995-865f-151b4a336c91-kube-api-access-f8zt8\") pod \"payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj\" (UID: \"9c96011c-d91f-4995-865f-151b4a336c91\") " pod="openshift-ingress/payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj" Oct 30 18:57:13.213831 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:13.213806 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c96011c-d91f-4995-865f-151b4a336c91-plugins-config-volume\") pod \"payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj\" (UID: \"9c96011c-d91f-4995-865f-151b4a336c91\") " pod="openshift-ingress/payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj" Oct 30 18:57:13.343596 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:13.343421 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:13.344403 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:13.343933 4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:57:13.344403 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:13.344003 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" podUID="ed1aa9fc-8605-4798-8b39-b75f2dde0337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 18:57:13.347170 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:13.344636 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8zt8\" (UniqueName: \"kubernetes.io/projected/9c96011c-d91f-4995-865f-151b4a336c91-kube-api-access-f8zt8\") pod \"payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj\" (UID: \"9c96011c-d91f-4995-865f-151b4a336c91\") " pod="openshift-ingress/payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj" Oct 30 18:57:13.347170 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:13.344762 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c96011c-d91f-4995-865f-151b4a336c91-plugins-config-volume\") pod \"payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj\" (UID: \"9c96011c-d91f-4995-865f-151b4a336c91\") " pod="openshift-ingress/payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj" Oct 30 18:57:13.347170 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:13.345940 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c96011c-d91f-4995-865f-151b4a336c91-plugins-config-volume\") pod \"payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj\" (UID: \"9c96011c-d91f-4995-865f-151b4a336c91\") " pod="openshift-ingress/payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj" Oct 30 18:57:13.350951 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:13.350924 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7"] Oct 30 18:57:13.356455 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:13.356431 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7" Oct 30 18:57:13.382925 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:13.382898 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8zt8\" (UniqueName: \"kubernetes.io/projected/9c96011c-d91f-4995-865f-151b4a336c91-kube-api-access-f8zt8\") pod \"payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj\" (UID: \"9c96011c-d91f-4995-865f-151b4a336c91\") " pod="openshift-ingress/payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj" Oct 30 18:57:13.387143 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:13.387100 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7"] Oct 30 18:57:13.405997 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:13.405967 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj" Oct 30 18:57:13.446079 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:13.446009 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2dd9790-3a6d-4928-8e56-972f41e9b9c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7\" (UID: \"b2dd9790-3a6d-4928-8e56-972f41e9b9c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7" Oct 30 18:57:13.446255 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:13.446116 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7c4d\" (UniqueName: \"kubernetes.io/projected/b2dd9790-3a6d-4928-8e56-972f41e9b9c1-kube-api-access-q7c4d\") pod \"payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7\" (UID: \"b2dd9790-3a6d-4928-8e56-972f41e9b9c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7" Oct 30 18:57:13.547897 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:13.547852 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2dd9790-3a6d-4928-8e56-972f41e9b9c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7\" (UID: \"b2dd9790-3a6d-4928-8e56-972f41e9b9c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7" Oct 30 18:57:13.548116 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:13.547945 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7c4d\" (UniqueName: \"kubernetes.io/projected/b2dd9790-3a6d-4928-8e56-972f41e9b9c1-kube-api-access-q7c4d\") pod \"payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7\" (UID: \"b2dd9790-3a6d-4928-8e56-972f41e9b9c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7" Oct 30 18:57:13.548944 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:13.548905 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2dd9790-3a6d-4928-8e56-972f41e9b9c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7\" (UID: \"b2dd9790-3a6d-4928-8e56-972f41e9b9c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7" Oct 30 18:57:13.609218 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:13.609143 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7c4d\" (UniqueName: \"kubernetes.io/projected/b2dd9790-3a6d-4928-8e56-972f41e9b9c1-kube-api-access-q7c4d\") pod \"payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7\" (UID: \"b2dd9790-3a6d-4928-8e56-972f41e9b9c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7" Oct 30 18:57:13.671772 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:13.671731 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7" Oct 30 18:57:13.848544 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:13.844102 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj"] Oct 30 18:57:13.851617 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:57:13.851585 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c96011c_d91f_4995_865f_151b4a336c91.slice/crio-b8b5cb54d87ab53e1ea34ac2f8bac159f99bb3be05b85904f3db9cb99a1b2110 WatchSource:0}: Error finding container b8b5cb54d87ab53e1ea34ac2f8bac159f99bb3be05b85904f3db9cb99a1b2110: Status 404 returned error can't find the container with id b8b5cb54d87ab53e1ea34ac2f8bac159f99bb3be05b85904f3db9cb99a1b2110 Oct 30 18:57:14.074763 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:14.074737 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7"] Oct 30 18:57:14.075254 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:57:14.075225 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2dd9790_3a6d_4928_8e56_972f41e9b9c1.slice/crio-d335455278483d7dd650140e8f9ae58891d5a73ea37b86f1aa118a406afec7aa WatchSource:0}: Error finding container d335455278483d7dd650140e8f9ae58891d5a73ea37b86f1aa118a406afec7aa: Status 404 returned error can't find the container with id d335455278483d7dd650140e8f9ae58891d5a73ea37b86f1aa118a406afec7aa Oct 30 18:57:14.334155 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:14.332527 4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:57:14.334155 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:14.332586 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" podUID="ed1aa9fc-8605-4798-8b39-b75f2dde0337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 18:57:14.372489 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:14.372455 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7" event={"ID":"b2dd9790-3a6d-4928-8e56-972f41e9b9c1","Type":"ContainerStarted","Data":"ae9667157b050795cb267c1cf48f93cf9883955e1fa0a35621e5a65181b8b897"} Oct 30 18:57:14.372690 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:14.372506 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7" event={"ID":"b2dd9790-3a6d-4928-8e56-972f41e9b9c1","Type":"ContainerStarted","Data":"d335455278483d7dd650140e8f9ae58891d5a73ea37b86f1aa118a406afec7aa"} Oct 30 18:57:14.372690 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:14.372534 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7" Oct 30 18:57:14.374185 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:14.374159 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj" event={"ID":"9c96011c-d91f-4995-865f-151b4a336c91","Type":"ContainerStarted","Data":"18f79b7dbc639cb88e07e8ccca89461da62e8065bc8acaf6f7961eeff5183706"} Oct 30 18:57:14.374321 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:14.374205 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj" event={"ID":"9c96011c-d91f-4995-865f-151b4a336c91","Type":"ContainerStarted","Data":"b8b5cb54d87ab53e1ea34ac2f8bac159f99bb3be05b85904f3db9cb99a1b2110"} Oct 30 18:57:14.374321 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:14.374300 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj" Oct 30 18:57:14.498699 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:14.498634 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7" podStartSLOduration=1.498615448 podStartE2EDuration="1.498615448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:14.497659533 +0000 UTC m=+1394.524764492" watchObservedRunningTime="2026-10-30 18:57:14.498615448 +0000 UTC m=+1394.525720411" Oct 30 18:57:14.623152 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:14.615563 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj" podStartSLOduration=1.615541098 podStartE2EDuration="1.615541098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:14.613029662 +0000 UTC m=+1394.640134621" watchObservedRunningTime="2026-10-30 18:57:14.615541098 +0000 UTC m=+1394.642646059" Oct 30 18:57:15.335158 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:15.329718 4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:57:15.335158 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:15.329779 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" podUID="ed1aa9fc-8605-4798-8b39-b75f2dde0337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 18:57:16.333154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:16.329662 4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:57:16.333154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:16.329753 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" podUID="ed1aa9fc-8605-4798-8b39-b75f2dde0337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 18:57:17.340188 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:17.335549 4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:57:17.340188 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:17.335614 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" podUID="ed1aa9fc-8605-4798-8b39-b75f2dde0337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 18:57:18.327338 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:18.327306 4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:57:18.327521 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:18.327355 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" podUID="ed1aa9fc-8605-4798-8b39-b75f2dde0337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 18:57:19.052790 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:19.052760 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e41ba5fe-nhxtk"] Oct 30 18:57:19.102274 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:19.102231 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e41ba5fe-nhxtk"] Oct 30 18:57:19.360522 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:19.360418 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:19.361539 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:19.361514 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:19.362766 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:19.362712 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 18:57:20.100895 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:20.100864 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70fb0b0c-7934-41b8-ae2a-5155338da606" path="/var/lib/kubelet/pods/70fb0b0c-7934-41b8-ae2a-5155338da606/volumes" Oct 30 18:57:25.384759 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:25.384726 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7" Oct 30 18:57:25.385309 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:25.384800 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj" Oct 30 18:57:26.399509 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:57:26.399466 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:57:28.049856 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:28.049815 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3e044f1-tctmr"] Oct 30 18:57:28.055473 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:28.055447 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e044f1-tctmr" Oct 30 18:57:28.111164 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:28.111115 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:57:28.158955 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:28.158923 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:57:28.158955 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:28.158951 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:57:28.159231 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:28.158951 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:57:28.163253 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:28.163221 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e26a1ca-1ae0-4609-bf0f-a366f9462bb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3e044f1-tctmr\" (UID: \"5e26a1ca-1ae0-4609-bf0f-a366f9462bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e044f1-tctmr" Oct 30 18:57:28.163411 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:28.163392 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e26a1ca-1ae0-4609-bf0f-a366f9462bb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3e044f1-tctmr\" (UID: \"5e26a1ca-1ae0-4609-bf0f-a366f9462bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e044f1-tctmr" Oct 30 18:57:28.163546 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:28.163525 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l2q4\" (UniqueName: \"kubernetes.io/projected/5e26a1ca-1ae0-4609-bf0f-a366f9462bb4-kube-api-access-9l2q4\") pod \"maas-api-revoke-sub-d3e044f1-tctmr\" (UID: \"5e26a1ca-1ae0-4609-bf0f-a366f9462bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e044f1-tctmr" Oct 30 18:57:28.165778 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:28.165750 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3e044f1-tctmr"] Oct 30 18:57:28.265194 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:28.265114 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e26a1ca-1ae0-4609-bf0f-a366f9462bb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3e044f1-tctmr\" (UID: \"5e26a1ca-1ae0-4609-bf0f-a366f9462bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e044f1-tctmr" Oct 30 18:57:28.265413 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:28.265284 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l2q4\" (UniqueName: \"kubernetes.io/projected/5e26a1ca-1ae0-4609-bf0f-a366f9462bb4-kube-api-access-9l2q4\") pod \"maas-api-revoke-sub-d3e044f1-tctmr\" (UID: \"5e26a1ca-1ae0-4609-bf0f-a366f9462bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e044f1-tctmr" Oct 30 18:57:28.265413 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:28.265358 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e26a1ca-1ae0-4609-bf0f-a366f9462bb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3e044f1-tctmr\" (UID: \"5e26a1ca-1ae0-4609-bf0f-a366f9462bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e044f1-tctmr" Oct 30 18:57:28.266018 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:28.265938 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e26a1ca-1ae0-4609-bf0f-a366f9462bb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3e044f1-tctmr\" (UID: \"5e26a1ca-1ae0-4609-bf0f-a366f9462bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e044f1-tctmr" Oct 30 18:57:28.267846 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:28.267797 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e26a1ca-1ae0-4609-bf0f-a366f9462bb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3e044f1-tctmr\" (UID: \"5e26a1ca-1ae0-4609-bf0f-a366f9462bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e044f1-tctmr" Oct 30 18:57:28.328220 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:28.328147 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l2q4\" (UniqueName: \"kubernetes.io/projected/5e26a1ca-1ae0-4609-bf0f-a366f9462bb4-kube-api-access-9l2q4\") pod \"maas-api-revoke-sub-d3e044f1-tctmr\" (UID: \"5e26a1ca-1ae0-4609-bf0f-a366f9462bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e044f1-tctmr" Oct 30 18:57:28.371203 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:28.371165 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e044f1-tctmr" Oct 30 18:57:28.754597 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:57:28.754567 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e26a1ca_1ae0_4609_bf0f_a366f9462bb4.slice/crio-f5cecbe19a2803fb11bb034c01ebdb5c453ca6a6b79e005e72e52d478f53aebc WatchSource:0}: Error finding container f5cecbe19a2803fb11bb034c01ebdb5c453ca6a6b79e005e72e52d478f53aebc: Status 404 returned error can't find the container with id f5cecbe19a2803fb11bb034c01ebdb5c453ca6a6b79e005e72e52d478f53aebc Oct 30 18:57:28.755271 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:28.755244 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3e044f1-tctmr"] Oct 30 18:57:29.497043 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:29.497008 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e044f1-tctmr" event={"ID":"5e26a1ca-1ae0-4609-bf0f-a366f9462bb4","Type":"ContainerStarted","Data":"821ddcf1e4e0ecf716153acce6b0d1a79c60c645a7e5b2872997b498ba5ec03e"} Oct 30 18:57:29.497584 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:29.497110 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e044f1-tctmr" event={"ID":"5e26a1ca-1ae0-4609-bf0f-a366f9462bb4","Type":"ContainerStarted","Data":"f5cecbe19a2803fb11bb034c01ebdb5c453ca6a6b79e005e72e52d478f53aebc"} Oct 30 18:57:29.615205 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:29.615100 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e044f1-tctmr" podStartSLOduration=2.615084244 podStartE2EDuration="2.615084244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:29.613278247 +0000 UTC m=+1409.640383206" watchObservedRunningTime="2026-10-30 18:57:29.615084244 +0000 UTC m=+1409.642189203" Oct 30 18:57:30.505557 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:30.505522 4531 generic.go:356] "Generic (PLEG): container finished" podID="5e26a1ca-1ae0-4609-bf0f-a366f9462bb4" containerID="821ddcf1e4e0ecf716153acce6b0d1a79c60c645a7e5b2872997b498ba5ec03e" exitCode=0 Oct 30 18:57:30.505956 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:30.505573 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e044f1-tctmr" event={"ID":"5e26a1ca-1ae0-4609-bf0f-a366f9462bb4","Type":"ContainerDied","Data":"821ddcf1e4e0ecf716153acce6b0d1a79c60c645a7e5b2872997b498ba5ec03e"} Oct 30 18:57:31.734003 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:31.733972 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e044f1-tctmr" Oct 30 18:57:31.912182 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:31.912066 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e26a1ca-1ae0-4609-bf0f-a366f9462bb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e26a1ca-1ae0-4609-bf0f-a366f9462bb4-maas-api-xks-ca\") pod \"5e26a1ca-1ae0-4609-bf0f-a366f9462bb4\" (UID: \"5e26a1ca-1ae0-4609-bf0f-a366f9462bb4\") " Oct 30 18:57:31.912366 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:31.912268 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e26a1ca-1ae0-4609-bf0f-a366f9462bb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e26a1ca-1ae0-4609-bf0f-a366f9462bb4-maas-api-openshift-ca\") pod \"5e26a1ca-1ae0-4609-bf0f-a366f9462bb4\" (UID: \"5e26a1ca-1ae0-4609-bf0f-a366f9462bb4\") " Oct 30 18:57:31.912438 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:31.912392 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e26a1ca-1ae0-4609-bf0f-a366f9462bb4-kube-api-access-9l2q4\" (UniqueName: \"kubernetes.io/projected/5e26a1ca-1ae0-4609-bf0f-a366f9462bb4-kube-api-access-9l2q4\") pod \"5e26a1ca-1ae0-4609-bf0f-a366f9462bb4\" (UID: \"5e26a1ca-1ae0-4609-bf0f-a366f9462bb4\") " Oct 30 18:57:31.913020 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:31.912663 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e26a1ca-1ae0-4609-bf0f-a366f9462bb4-maas-api-openshift-ca" pod "5e26a1ca-1ae0-4609-bf0f-a366f9462bb4" (UID: "5e26a1ca-1ae0-4609-bf0f-a366f9462bb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:31.913020 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:31.912975 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e26a1ca-1ae0-4609-bf0f-a366f9462bb4-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:57:31.915357 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:31.915319 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e26a1ca-1ae0-4609-bf0f-a366f9462bb4-kube-api-access-9l2q4" pod "5e26a1ca-1ae0-4609-bf0f-a366f9462bb4" (UID: "5e26a1ca-1ae0-4609-bf0f-a366f9462bb4"). InnerVolumeSpecName "kube-api-access-9l2q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:31.916507 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:31.916480 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e26a1ca-1ae0-4609-bf0f-a366f9462bb4-maas-api-xks-ca" pod "5e26a1ca-1ae0-4609-bf0f-a366f9462bb4" (UID: "5e26a1ca-1ae0-4609-bf0f-a366f9462bb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:32.014653 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:32.014613 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l2q4\" (UniqueName: \"kubernetes.io/projected/5e26a1ca-1ae0-4609-bf0f-a366f9462bb4-kube-api-access-9l2q4\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:57:32.014653 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:32.014647 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e26a1ca-1ae0-4609-bf0f-a366f9462bb4-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:57:32.524955 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:32.524924 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e044f1-tctmr" Oct 30 18:57:32.525189 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:32.524959 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3e044f1-tctmr" event={"ID":"5e26a1ca-1ae0-4609-bf0f-a366f9462bb4","Type":"ContainerDied","Data":"f5cecbe19a2803fb11bb034c01ebdb5c453ca6a6b79e005e72e52d478f53aebc"} Oct 30 18:57:32.525189 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:32.525054 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5cecbe19a2803fb11bb034c01ebdb5c453ca6a6b79e005e72e52d478f53aebc" Oct 30 18:57:37.152435 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.152397 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp"] Oct 30 18:57:37.153970 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.153943 4531 cpu_manager.go:414] "containerMap: removing container" podUID="5e26a1ca-1ae0-4609-bf0f-a366f9462bb4" containerName="revoke-keys" Oct 30 18:57:37.154156 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.154121 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e26a1ca-1ae0-4609-bf0f-a366f9462bb4" containerName="revoke-keys" Oct 30 18:57:37.154447 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.154432 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e26a1ca-1ae0-4609-bf0f-a366f9462bb4" containerName="revoke-keys" Oct 30 18:57:37.164221 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.164107 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.218793 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.217471 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-470093-openshift-default-dockercfg-l97zt\"" Oct 30 18:57:37.283906 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.283863 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-workload-socket\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.284095 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.283943 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-istio-data\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.284095 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.284053 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-credential-socket\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.284242 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.284101 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90016c74-97f8-4976-87c0-1603b897c2f3-istiod-ca-cert\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.284242 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.284223 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90016c74-97f8-4976-87c0-1603b897c2f3-istio-podinfo\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.284470 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.284444 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-workload-certs\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.284593 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.284484 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-istio-envoy\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.284593 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.284507 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90016c74-97f8-4976-87c0-1603b897c2f3-istio-token\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.284593 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.284572 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hkw8\" (UniqueName: \"kubernetes.io/projected/90016c74-97f8-4976-87c0-1603b897c2f3-kube-api-access-7hkw8\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.297577 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.297543 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp"] Oct 30 18:57:37.386330 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.386293 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-workload-certs\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.386330 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.386333 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-istio-envoy\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.386592 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.386350 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90016c74-97f8-4976-87c0-1603b897c2f3-istio-token\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.386592 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.386374 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hkw8\" (UniqueName: \"kubernetes.io/projected/90016c74-97f8-4976-87c0-1603b897c2f3-kube-api-access-7hkw8\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.386592 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.386448 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-workload-socket\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.386592 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.386504 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-istio-data\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.386827 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.386750 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-workload-certs\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.386885 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.386863 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-workload-socket\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.386940 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.386885 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-credential-socket\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.386940 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.386919 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90016c74-97f8-4976-87c0-1603b897c2f3-istiod-ca-cert\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.387048 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.386939 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-istio-data\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.387048 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.386947 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90016c74-97f8-4976-87c0-1603b897c2f3-istio-podinfo\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.387258 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.387235 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-credential-socket\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.388234 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.388207 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90016c74-97f8-4976-87c0-1603b897c2f3-istiod-ca-cert\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.388963 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.388939 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-istio-envoy\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.389621 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.389600 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90016c74-97f8-4976-87c0-1603b897c2f3-istio-podinfo\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.460514 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.460444 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hkw8\" (UniqueName: \"kubernetes.io/projected/90016c74-97f8-4976-87c0-1603b897c2f3-kube-api-access-7hkw8\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.466962 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.466932 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90016c74-97f8-4976-87c0-1603b897c2f3-istio-token\") pod \"e2e-api-b-470093-openshift-default-6745c4765-8xfqp\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.525564 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.525526 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:37.975116 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:37.975078 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp"] Oct 30 18:57:38.010259 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:57:38.010225 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90016c74_97f8_4976_87c0_1603b897c2f3.slice/crio-2a3a1f28ad651e8be5a7795c96621728c3b6bcfbd45b0b452e89e0134951d21d WatchSource:0}: Error finding container 2a3a1f28ad651e8be5a7795c96621728c3b6bcfbd45b0b452e89e0134951d21d: Status 404 returned error can't find the container with id 2a3a1f28ad651e8be5a7795c96621728c3b6bcfbd45b0b452e89e0134951d21d Oct 30 18:57:38.012897 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:38.012865 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:57:38.012965 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:38.012936 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:57:38.013007 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:38.012965 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:57:38.572852 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:38.572807 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" event={"ID":"90016c74-97f8-4976-87c0-1603b897c2f3","Type":"ContainerStarted","Data":"595c6a03ff16516ae450f50452663d04111f911a3c4c9d0a5f4586f5f88f9c14"} Oct 30 18:57:38.573355 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:38.572863 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" event={"ID":"90016c74-97f8-4976-87c0-1603b897c2f3","Type":"ContainerStarted","Data":"2a3a1f28ad651e8be5a7795c96621728c3b6bcfbd45b0b452e89e0134951d21d"} Oct 30 18:57:38.696322 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:38.696269 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" podStartSLOduration=1.6962552450000001 podStartE2EDuration="1.696255245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:38.692806823 +0000 UTC m=+1418.719911783" watchObservedRunningTime="2026-10-30 18:57:38.696255245 +0000 UTC m=+1418.723360204" Oct 30 18:57:39.525763 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:39.525726 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:39.531538 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:39.531514 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:39.581273 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:39.581096 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:39.582708 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:39.582686 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:57:45.857303 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:45.857265 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5"] Oct 30 18:57:45.862898 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:45.862873 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5" Oct 30 18:57:45.925750 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:45.925712 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5"] Oct 30 18:57:45.930122 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:45.930080 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcbp9\" (UniqueName: \"kubernetes.io/projected/8974a32e-1a13-435a-b4cc-1948881c3532-kube-api-access-jcbp9\") pod \"payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5\" (UID: \"8974a32e-1a13-435a-b4cc-1948881c3532\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5" Oct 30 18:57:45.930818 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:45.930785 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8974a32e-1a13-435a-b4cc-1948881c3532-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5\" (UID: \"8974a32e-1a13-435a-b4cc-1948881c3532\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5" Oct 30 18:57:45.973485 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:45.973449 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-470093\"" Oct 30 18:57:46.044589 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:46.044550 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8974a32e-1a13-435a-b4cc-1948881c3532-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5\" (UID: \"8974a32e-1a13-435a-b4cc-1948881c3532\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5" Oct 30 18:57:46.044762 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:46.044699 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcbp9\" (UniqueName: \"kubernetes.io/projected/8974a32e-1a13-435a-b4cc-1948881c3532-kube-api-access-jcbp9\") pod \"payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5\" (UID: \"8974a32e-1a13-435a-b4cc-1948881c3532\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5" Oct 30 18:57:46.046566 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:46.045938 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8974a32e-1a13-435a-b4cc-1948881c3532-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5\" (UID: \"8974a32e-1a13-435a-b4cc-1948881c3532\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5" Oct 30 18:57:46.110495 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:46.110403 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcbp9\" (UniqueName: \"kubernetes.io/projected/8974a32e-1a13-435a-b4cc-1948881c3532-kube-api-access-jcbp9\") pod \"payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5\" (UID: \"8974a32e-1a13-435a-b4cc-1948881c3532\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5" Oct 30 18:57:46.179202 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:46.179161 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5" Oct 30 18:57:46.584245 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:46.584219 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5"] Oct 30 18:57:46.584916 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:57:46.584887 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8974a32e_1a13_435a_b4cc_1948881c3532.slice/crio-6869226c0c56434b8ffafa23d3ed12b3f5fed1af4648f1354462cdf14e026e44 WatchSource:0}: Error finding container 6869226c0c56434b8ffafa23d3ed12b3f5fed1af4648f1354462cdf14e026e44: Status 404 returned error can't find the container with id 6869226c0c56434b8ffafa23d3ed12b3f5fed1af4648f1354462cdf14e026e44 Oct 30 18:57:46.633017 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:46.632984 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5" event={"ID":"8974a32e-1a13-435a-b4cc-1948881c3532","Type":"ContainerStarted","Data":"6869226c0c56434b8ffafa23d3ed12b3f5fed1af4648f1354462cdf14e026e44"} Oct 30 18:57:47.644402 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:47.644366 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5" event={"ID":"8974a32e-1a13-435a-b4cc-1948881c3532","Type":"ContainerStarted","Data":"27476dec085c2c4c97a6fd27462fd641861bbd5c084164965e37595d58f2d1fc"} Oct 30 18:57:47.644825 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:47.644505 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5" Oct 30 18:57:47.772150 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:47.772076 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5" podStartSLOduration=2.772061248 podStartE2EDuration="2.772061248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:47.770816067 +0000 UTC m=+1427.797921025" watchObservedRunningTime="2026-10-30 18:57:47.772061248 +0000 UTC m=+1427.799166206" Oct 30 18:57:53.600938 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:53.600900 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j"] Oct 30 18:57:53.605592 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:53.605570 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j" Oct 30 18:57:53.664332 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:53.664299 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-83d5bd\"" Oct 30 18:57:53.665351 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:53.665327 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-83d5bd-dockercfg-jpchz\"" Oct 30 18:57:53.683108 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:53.683009 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j"] Oct 30 18:57:53.743292 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:53.743252 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a58321dd-1bd5-45e5-89d1-1a9a60aaadf9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j\" (UID: \"a58321dd-1bd5-45e5-89d1-1a9a60aaadf9\") " pod="openshift-ingress/payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j" Oct 30 18:57:53.743630 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:53.743601 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7hts\" (UniqueName: \"kubernetes.io/projected/a58321dd-1bd5-45e5-89d1-1a9a60aaadf9-kube-api-access-v7hts\") pod \"payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j\" (UID: \"a58321dd-1bd5-45e5-89d1-1a9a60aaadf9\") " pod="openshift-ingress/payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j" Oct 30 18:57:53.778301 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:53.778264 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx"] Oct 30 18:57:53.782969 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:53.782944 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx" Oct 30 18:57:53.841052 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:53.841023 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx"] Oct 30 18:57:53.845122 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:53.845090 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7hts\" (UniqueName: \"kubernetes.io/projected/a58321dd-1bd5-45e5-89d1-1a9a60aaadf9-kube-api-access-v7hts\") pod \"payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j\" (UID: \"a58321dd-1bd5-45e5-89d1-1a9a60aaadf9\") " pod="openshift-ingress/payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j" Oct 30 18:57:53.845334 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:53.845210 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a58321dd-1bd5-45e5-89d1-1a9a60aaadf9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j\" (UID: \"a58321dd-1bd5-45e5-89d1-1a9a60aaadf9\") " pod="openshift-ingress/payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j" Oct 30 18:57:53.846118 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:53.846088 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a58321dd-1bd5-45e5-89d1-1a9a60aaadf9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j\" (UID: \"a58321dd-1bd5-45e5-89d1-1a9a60aaadf9\") " pod="openshift-ingress/payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j" Oct 30 18:57:53.911054 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:53.910981 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7hts\" (UniqueName: \"kubernetes.io/projected/a58321dd-1bd5-45e5-89d1-1a9a60aaadf9-kube-api-access-v7hts\") pod \"payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j\" (UID: \"a58321dd-1bd5-45e5-89d1-1a9a60aaadf9\") " pod="openshift-ingress/payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j" Oct 30 18:57:53.936363 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:53.936319 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j" Oct 30 18:57:53.949154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:53.947113 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dt4b\" (UniqueName: \"kubernetes.io/projected/a860f1c1-b883-44ba-bc53-e20c606ae368-kube-api-access-4dt4b\") pod \"payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx\" (UID: \"a860f1c1-b883-44ba-bc53-e20c606ae368\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx" Oct 30 18:57:53.949154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:53.947322 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a860f1c1-b883-44ba-bc53-e20c606ae368-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx\" (UID: \"a860f1c1-b883-44ba-bc53-e20c606ae368\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx" Oct 30 18:57:54.048201 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:54.048163 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dt4b\" (UniqueName: \"kubernetes.io/projected/a860f1c1-b883-44ba-bc53-e20c606ae368-kube-api-access-4dt4b\") pod \"payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx\" (UID: \"a860f1c1-b883-44ba-bc53-e20c606ae368\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx" Oct 30 18:57:54.048380 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:54.048364 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a860f1c1-b883-44ba-bc53-e20c606ae368-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx\" (UID: \"a860f1c1-b883-44ba-bc53-e20c606ae368\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx" Oct 30 18:57:54.049038 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:54.049011 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a860f1c1-b883-44ba-bc53-e20c606ae368-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx\" (UID: \"a860f1c1-b883-44ba-bc53-e20c606ae368\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx" Oct 30 18:57:54.109693 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:54.109654 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dt4b\" (UniqueName: \"kubernetes.io/projected/a860f1c1-b883-44ba-bc53-e20c606ae368-kube-api-access-4dt4b\") pod \"payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx\" (UID: \"a860f1c1-b883-44ba-bc53-e20c606ae368\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx" Oct 30 18:57:54.315066 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:54.315030 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j"] Oct 30 18:57:54.315964 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:57:54.315937 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda58321dd_1bd5_45e5_89d1_1a9a60aaadf9.slice/crio-618fc7f512602fabfa8efd748e721cbea0fc84c7ea2cd5a11f935ba1829652a5 WatchSource:0}: Error finding container 618fc7f512602fabfa8efd748e721cbea0fc84c7ea2cd5a11f935ba1829652a5: Status 404 returned error can't find the container with id 618fc7f512602fabfa8efd748e721cbea0fc84c7ea2cd5a11f935ba1829652a5 Oct 30 18:57:54.397201 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:54.397174 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx" Oct 30 18:57:54.725490 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:54.725405 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j" event={"ID":"a58321dd-1bd5-45e5-89d1-1a9a60aaadf9","Type":"ContainerStarted","Data":"75229c19a319013e42870d3c90b5ffc2138279a4819203c206d3eb8712fc2f31"} Oct 30 18:57:54.725883 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:54.725497 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j" event={"ID":"a58321dd-1bd5-45e5-89d1-1a9a60aaadf9","Type":"ContainerStarted","Data":"618fc7f512602fabfa8efd748e721cbea0fc84c7ea2cd5a11f935ba1829652a5"} Oct 30 18:57:54.725883 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:54.725637 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j" Oct 30 18:57:54.772976 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:54.772949 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx"] Oct 30 18:57:54.773502 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:57:54.773479 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda860f1c1_b883_44ba_bc53_e20c606ae368.slice/crio-afe58f875259ae2998a1c7cc2c6b75f7a4890205b86c49369f40d028faa6281d WatchSource:0}: Error finding container afe58f875259ae2998a1c7cc2c6b75f7a4890205b86c49369f40d028faa6281d: Status 404 returned error can't find the container with id afe58f875259ae2998a1c7cc2c6b75f7a4890205b86c49369f40d028faa6281d Oct 30 18:57:54.847416 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:54.847358 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j" podStartSLOduration=1.847342193 podStartE2EDuration="1.847342193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:54.84266027 +0000 UTC m=+1434.869765231" watchObservedRunningTime="2026-10-30 18:57:54.847342193 +0000 UTC m=+1434.874447165" Oct 30 18:57:55.734471 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:55.734423 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx" event={"ID":"a860f1c1-b883-44ba-bc53-e20c606ae368","Type":"ContainerStarted","Data":"df4f7b6ecc720ac10e78d5a18b2c10e44cdc45f746c5d370c37d37ff88ab1716"} Oct 30 18:57:55.734970 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:55.734484 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx" event={"ID":"a860f1c1-b883-44ba-bc53-e20c606ae368","Type":"ContainerStarted","Data":"afe58f875259ae2998a1c7cc2c6b75f7a4890205b86c49369f40d028faa6281d"} Oct 30 18:57:55.734970 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:55.734520 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx" Oct 30 18:57:55.850066 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:55.850007 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx" podStartSLOduration=2.849991778 podStartE2EDuration="2.849991778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:55.849450802 +0000 UTC m=+1435.876555766" watchObservedRunningTime="2026-10-30 18:57:55.849991778 +0000 UTC m=+1435.877096737" Oct 30 18:57:56.399071 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:57:56.399035 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:57:58.051189 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:58.051147 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e444fdb-qpbb4"] Oct 30 18:57:58.101064 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:58.101034 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e444fdb-qpbb4"] Oct 30 18:57:58.654728 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:57:58.654702 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5" Oct 30 18:58:00.100565 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:00.100529 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04e19546-630e-4c22-8b60-a21bb614c607" path="/var/lib/kubelet/pods/04e19546-630e-4c22-8b60-a21bb614c607/volumes" Oct 30 18:58:03.091169 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:03.081873 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09ff2cd1-nfggg"] Oct 30 18:58:03.109592 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:03.109554 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09ff2cd1-nfggg"] Oct 30 18:58:04.099968 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:04.099925 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="392fb7ab-dcd4-4b8b-846f-9d94f5b72f73" path="/var/lib/kubelet/pods/392fb7ab-dcd4-4b8b-846f-9d94f5b72f73/volumes" Oct 30 18:58:05.738327 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:05.738299 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j" Oct 30 18:58:06.744412 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:06.744376 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx" Oct 30 18:58:13.941975 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:13.941945 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7112-22348d"] Oct 30 18:58:13.946197 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:13.946180 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7112-22348d" Oct 30 18:58:14.002873 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:14.002837 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7112-22348d"] Oct 30 18:58:14.052161 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:14.052101 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlz74\" (UniqueName: \"kubernetes.io/projected/2d3558b9-24b0-45d9-b5b0-295b679b3662-kube-api-access-jlz74\") pod \"test-curl-7112-22348d\" (UID: \"2d3558b9-24b0-45d9-b5b0-295b679b3662\") " pod="opendatahub/test-curl-7112-22348d" Oct 30 18:58:14.127731 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:14.127700 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7q8sq\"" Oct 30 18:58:14.132233 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:14.132193 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d7196a3-fftps"] Oct 30 18:58:14.135832 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:14.135811 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d7196a3-fftps"] Oct 30 18:58:14.140356 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:14.140328 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a4e1823-jg6p8"] Oct 30 18:58:14.153252 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:14.153220 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlz74\" (UniqueName: \"kubernetes.io/projected/2d3558b9-24b0-45d9-b5b0-295b679b3662-kube-api-access-jlz74\") pod \"test-curl-7112-22348d\" (UID: \"2d3558b9-24b0-45d9-b5b0-295b679b3662\") " pod="opendatahub/test-curl-7112-22348d" Oct 30 18:58:14.181405 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:14.181352 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a4e1823-jg6p8"] Oct 30 18:58:14.232360 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:14.232277 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlz74\" (UniqueName: \"kubernetes.io/projected/2d3558b9-24b0-45d9-b5b0-295b679b3662-kube-api-access-jlz74\") pod \"test-curl-7112-22348d\" (UID: \"2d3558b9-24b0-45d9-b5b0-295b679b3662\") " pod="opendatahub/test-curl-7112-22348d" Oct 30 18:58:14.258633 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:14.258592 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7112-22348d" Oct 30 18:58:14.638090 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:14.638059 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7112-22348d"] Oct 30 18:58:14.638893 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:58:14.638864 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d3558b9_24b0_45d9_b5b0_295b679b3662.slice/crio-b478e50b693c3a1febda730a9ec0cd3d4b41443b48ab960b896187f4f5375256 WatchSource:0}: Error finding container b478e50b693c3a1febda730a9ec0cd3d4b41443b48ab960b896187f4f5375256: Status 404 returned error can't find the container with id b478e50b693c3a1febda730a9ec0cd3d4b41443b48ab960b896187f4f5375256 Oct 30 18:58:14.640814 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:14.640797 4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:58:14.880480 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:14.880445 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7112-22348d" event={"ID":"2d3558b9-24b0-45d9-b5b0-295b679b3662","Type":"ContainerStarted","Data":"b478e50b693c3a1febda730a9ec0cd3d4b41443b48ab960b896187f4f5375256"} Oct 30 18:58:16.100323 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:16.100290 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f9b77fb-c20a-4e51-96be-3e6580a8dc5d" path="/var/lib/kubelet/pods/0f9b77fb-c20a-4e51-96be-3e6580a8dc5d/volumes" Oct 30 18:58:16.100827 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:16.100811 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df118305-439d-446d-8175-2f7894fc6302" path="/var/lib/kubelet/pods/df118305-439d-446d-8175-2f7894fc6302/volumes" Oct 30 18:58:18.914804 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:18.914763 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7112-22348d" event={"ID":"2d3558b9-24b0-45d9-b5b0-295b679b3662","Type":"ContainerStarted","Data":"925d441348fe5d7772876c3dbc6813f7a9e44541a7e10caa875424b2c726408c"} Oct 30 18:58:19.034818 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:19.034750 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7112-22348d" podStartSLOduration=2.795392577 podStartE2EDuration="6.034731647s" totalImagesPullingTime="3.23933907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:19.031993096 +0000 UTC m=+1459.059098055" watchObservedRunningTime="2026-10-30 18:58:19.034731647 +0000 UTC m=+1459.061836606" Oct 30 18:58:20.697417 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:20.697373 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7112-22348d"] Oct 30 18:58:20.715494 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:20.715456 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7112-22348d"] Oct 30 18:58:20.929811 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:20.929763 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7112-22348d" podUID="2d3558b9-24b0-45d9-b5b0-295b679b3662" containerName="test-curl-7112-22348d" containerID="cri-o://925d441348fe5d7772876c3dbc6813f7a9e44541a7e10caa875424b2c726408c" gracePeriod=2 Oct 30 18:58:20.981832 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:58:20.981741 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7112-22348d\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="2d3558b9-24b0-45d9-b5b0-295b679b3662" pod="opendatahub/test-curl-7112-22348d" Oct 30 18:58:21.736521 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:21.736475 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm"] Oct 30 18:58:21.737039 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:21.737024 4531 cpu_manager.go:414] "containerMap: removing container" podUID="2d3558b9-24b0-45d9-b5b0-295b679b3662" containerName="test-curl-7112-22348d" Oct 30 18:58:21.737088 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:21.737040 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d3558b9-24b0-45d9-b5b0-295b679b3662" containerName="test-curl-7112-22348d" Oct 30 18:58:21.737186 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:21.737176 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d3558b9-24b0-45d9-b5b0-295b679b3662" containerName="test-curl-7112-22348d" Oct 30 18:58:21.740778 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:21.740753 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm" Oct 30 18:58:21.751561 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:21.751527 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92faba62-c6d6-4fc7-9ef2-8680f2c518eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm\" (UID: \"92faba62-c6d6-4fc7-9ef2-8680f2c518eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm" Oct 30 18:58:21.751704 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:21.751581 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92faba62-c6d6-4fc7-9ef2-8680f2c518eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm\" (UID: \"92faba62-c6d6-4fc7-9ef2-8680f2c518eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm" Oct 30 18:58:21.752010 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:21.751754 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw2ns\" (UniqueName: \"kubernetes.io/projected/92faba62-c6d6-4fc7-9ef2-8680f2c518eb-kube-api-access-bw2ns\") pod \"maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm\" (UID: \"92faba62-c6d6-4fc7-9ef2-8680f2c518eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm" Oct 30 18:58:21.795265 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:58:21.795222 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7112-22348d\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="2d3558b9-24b0-45d9-b5b0-295b679b3662" pod="opendatahub/test-curl-7112-22348d" Oct 30 18:58:21.797079 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:21.797057 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:58:21.805822 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:21.805776 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm"] Oct 30 18:58:21.844951 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:21.844919 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:58:21.846608 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:21.846581 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:58:21.846739 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:21.846587 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:58:21.853509 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:21.853474 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92faba62-c6d6-4fc7-9ef2-8680f2c518eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm\" (UID: \"92faba62-c6d6-4fc7-9ef2-8680f2c518eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm" Oct 30 18:58:21.853699 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:21.853522 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92faba62-c6d6-4fc7-9ef2-8680f2c518eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm\" (UID: \"92faba62-c6d6-4fc7-9ef2-8680f2c518eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm" Oct 30 18:58:21.853699 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:21.853608 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw2ns\" (UniqueName: \"kubernetes.io/projected/92faba62-c6d6-4fc7-9ef2-8680f2c518eb-kube-api-access-bw2ns\") pod \"maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm\" (UID: \"92faba62-c6d6-4fc7-9ef2-8680f2c518eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm" Oct 30 18:58:21.854411 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:21.854386 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92faba62-c6d6-4fc7-9ef2-8680f2c518eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm\" (UID: \"92faba62-c6d6-4fc7-9ef2-8680f2c518eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm" Oct 30 18:58:21.856371 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:21.856344 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92faba62-c6d6-4fc7-9ef2-8680f2c518eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm\" (UID: \"92faba62-c6d6-4fc7-9ef2-8680f2c518eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm" Oct 30 18:58:21.916286 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:21.916243 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw2ns\" (UniqueName: \"kubernetes.io/projected/92faba62-c6d6-4fc7-9ef2-8680f2c518eb-kube-api-access-bw2ns\") pod \"maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm\" (UID: \"92faba62-c6d6-4fc7-9ef2-8680f2c518eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm" Oct 30 18:58:22.055924 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:22.055886 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm" Oct 30 18:58:22.439451 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:22.439414 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm"] Oct 30 18:58:22.439609 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:58:22.439506 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92faba62_c6d6_4fc7_9ef2_8680f2c518eb.slice/crio-24dd9dafbc1312d7e1448f4edc548b652217e6048a319d02731cf910e86d2dfa WatchSource:0}: Error finding container 24dd9dafbc1312d7e1448f4edc548b652217e6048a319d02731cf910e86d2dfa: Status 404 returned error can't find the container with id 24dd9dafbc1312d7e1448f4edc548b652217e6048a319d02731cf910e86d2dfa Oct 30 18:58:22.946707 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:22.946661 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm" event={"ID":"92faba62-c6d6-4fc7-9ef2-8680f2c518eb","Type":"ContainerStarted","Data":"2e3eac8993e944b0f77e00199f578421f96a1acf54c5dc25c401384a292b8551"} Oct 30 18:58:22.947092 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:22.946767 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm" event={"ID":"92faba62-c6d6-4fc7-9ef2-8680f2c518eb","Type":"ContainerStarted","Data":"24dd9dafbc1312d7e1448f4edc548b652217e6048a319d02731cf910e86d2dfa"} Oct 30 18:58:23.067830 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:23.067768 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm" podStartSLOduration=2.067752606 podStartE2EDuration="2.067752606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:23.066232428 +0000 UTC m=+1463.093337391" watchObservedRunningTime="2026-10-30 18:58:23.067752606 +0000 UTC m=+1463.094857565" Oct 30 18:58:23.183591 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:23.183552 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7112-22348d" Oct 30 18:58:23.269766 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:23.269723 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d3558b9-24b0-45d9-b5b0-295b679b3662-kube-api-access-jlz74\" (UniqueName: \"kubernetes.io/projected/2d3558b9-24b0-45d9-b5b0-295b679b3662-kube-api-access-jlz74\") pod \"2d3558b9-24b0-45d9-b5b0-295b679b3662\" (UID: \"2d3558b9-24b0-45d9-b5b0-295b679b3662\") " Oct 30 18:58:23.272063 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:23.272030 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d3558b9-24b0-45d9-b5b0-295b679b3662-kube-api-access-jlz74" pod "2d3558b9-24b0-45d9-b5b0-295b679b3662" (UID: "2d3558b9-24b0-45d9-b5b0-295b679b3662"). InnerVolumeSpecName "kube-api-access-jlz74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:23.371321 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:23.371219 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlz74\" (UniqueName: \"kubernetes.io/projected/2d3558b9-24b0-45d9-b5b0-295b679b3662-kube-api-access-jlz74\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:58:23.955290 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:23.955247 4531 generic.go:356] "Generic (PLEG): container finished" podID="2d3558b9-24b0-45d9-b5b0-295b679b3662" containerID="925d441348fe5d7772876c3dbc6813f7a9e44541a7e10caa875424b2c726408c" exitCode=137 Oct 30 18:58:23.955777 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:23.955297 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7112-22348d" Oct 30 18:58:23.955777 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:23.955343 4531 scope.go:122] "RemoveContainer" containerID="925d441348fe5d7772876c3dbc6813f7a9e44541a7e10caa875424b2c726408c" Oct 30 18:58:23.957626 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:23.957587 4531 generic.go:356] "Generic (PLEG): container finished" podID="92faba62-c6d6-4fc7-9ef2-8680f2c518eb" containerID="2e3eac8993e944b0f77e00199f578421f96a1acf54c5dc25c401384a292b8551" exitCode=0 Oct 30 18:58:23.957795 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:23.957674 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm" event={"ID":"92faba62-c6d6-4fc7-9ef2-8680f2c518eb","Type":"ContainerDied","Data":"2e3eac8993e944b0f77e00199f578421f96a1acf54c5dc25c401384a292b8551"} Oct 30 18:58:23.967042 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:23.967016 4531 scope.go:122] "RemoveContainer" containerID="925d441348fe5d7772876c3dbc6813f7a9e44541a7e10caa875424b2c726408c" Oct 30 18:58:23.967476 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:58:23.967451 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"925d441348fe5d7772876c3dbc6813f7a9e44541a7e10caa875424b2c726408c\": container with ID starting with 925d441348fe5d7772876c3dbc6813f7a9e44541a7e10caa875424b2c726408c not found: ID does not exist" containerID="925d441348fe5d7772876c3dbc6813f7a9e44541a7e10caa875424b2c726408c" Oct 30 18:58:23.967533 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:23.967486 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"925d441348fe5d7772876c3dbc6813f7a9e44541a7e10caa875424b2c726408c"} err="failed to get container status \"925d441348fe5d7772876c3dbc6813f7a9e44541a7e10caa875424b2c726408c\": rpc error: code = NotFound desc = could not find container \"925d441348fe5d7772876c3dbc6813f7a9e44541a7e10caa875424b2c726408c\": container with ID starting with 925d441348fe5d7772876c3dbc6813f7a9e44541a7e10caa875424b2c726408c not found: ID does not exist" Oct 30 18:58:24.102341 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:24.102309 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d3558b9-24b0-45d9-b5b0-295b679b3662" path="/var/lib/kubelet/pods/2d3558b9-24b0-45d9-b5b0-295b679b3662/volumes" Oct 30 18:58:24.143045 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:58:24.142956 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7112-22348d\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="2d3558b9-24b0-45d9-b5b0-295b679b3662" pod="opendatahub/test-curl-7112-22348d" Oct 30 18:58:25.132682 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:25.132645 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm" Oct 30 18:58:25.196274 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:25.196238 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92faba62-c6d6-4fc7-9ef2-8680f2c518eb-kube-api-access-bw2ns\" (UniqueName: \"kubernetes.io/projected/92faba62-c6d6-4fc7-9ef2-8680f2c518eb-kube-api-access-bw2ns\") pod \"92faba62-c6d6-4fc7-9ef2-8680f2c518eb\" (UID: \"92faba62-c6d6-4fc7-9ef2-8680f2c518eb\") " Oct 30 18:58:25.196468 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:25.196418 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92faba62-c6d6-4fc7-9ef2-8680f2c518eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92faba62-c6d6-4fc7-9ef2-8680f2c518eb-maas-api-openshift-ca\") pod \"92faba62-c6d6-4fc7-9ef2-8680f2c518eb\" (UID: \"92faba62-c6d6-4fc7-9ef2-8680f2c518eb\") " Oct 30 18:58:25.196527 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:25.196509 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92faba62-c6d6-4fc7-9ef2-8680f2c518eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92faba62-c6d6-4fc7-9ef2-8680f2c518eb-maas-api-xks-ca\") pod \"92faba62-c6d6-4fc7-9ef2-8680f2c518eb\" (UID: \"92faba62-c6d6-4fc7-9ef2-8680f2c518eb\") " Oct 30 18:58:25.196826 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:25.196798 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92faba62-c6d6-4fc7-9ef2-8680f2c518eb-maas-api-openshift-ca" pod "92faba62-c6d6-4fc7-9ef2-8680f2c518eb" (UID: "92faba62-c6d6-4fc7-9ef2-8680f2c518eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:25.197245 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:25.197209 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92faba62-c6d6-4fc7-9ef2-8680f2c518eb-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:58:25.198650 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:25.198624 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92faba62-c6d6-4fc7-9ef2-8680f2c518eb-kube-api-access-bw2ns" pod "92faba62-c6d6-4fc7-9ef2-8680f2c518eb" (UID: "92faba62-c6d6-4fc7-9ef2-8680f2c518eb"). InnerVolumeSpecName "kube-api-access-bw2ns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:25.198752 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:25.198721 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92faba62-c6d6-4fc7-9ef2-8680f2c518eb-maas-api-xks-ca" pod "92faba62-c6d6-4fc7-9ef2-8680f2c518eb" (UID: "92faba62-c6d6-4fc7-9ef2-8680f2c518eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:25.299076 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:25.299035 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92faba62-c6d6-4fc7-9ef2-8680f2c518eb-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:58:25.299076 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:25.299076 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw2ns\" (UniqueName: \"kubernetes.io/projected/92faba62-c6d6-4fc7-9ef2-8680f2c518eb-kube-api-access-bw2ns\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:58:25.974526 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:25.974492 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm" Oct 30 18:58:25.974736 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:25.974519 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm" event={"ID":"92faba62-c6d6-4fc7-9ef2-8680f2c518eb","Type":"ContainerDied","Data":"24dd9dafbc1312d7e1448f4edc548b652217e6048a319d02731cf910e86d2dfa"} Oct 30 18:58:25.974736 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:25.974571 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24dd9dafbc1312d7e1448f4edc548b652217e6048a319d02731cf910e86d2dfa" Oct 30 18:58:26.401342 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:58:26.401306 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:58:31.098800 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:31.098757 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf4f477c-5c9jv"] Oct 30 18:58:31.156786 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:31.156744 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf4f477c-5c9jv"] Oct 30 18:58:32.064773 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:32.064738 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b161d939-k5xqz"] Oct 30 18:58:32.106538 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:32.106504 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd8446ae-527e-451f-81f6-88f7afd3f457" path="/var/lib/kubelet/pods/bd8446ae-527e-451f-81f6-88f7afd3f457/volumes" Oct 30 18:58:32.111791 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:32.111764 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b161d939-k5xqz"] Oct 30 18:58:33.284417 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:33.284382 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k"] Oct 30 18:58:33.872333 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:33.872296 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2730df4c-5q8g7"] Oct 30 18:58:33.872964 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:33.872944 4531 cpu_manager.go:414] "containerMap: removing container" podUID="92faba62-c6d6-4fc7-9ef2-8680f2c518eb" containerName="revoke-keys" Oct 30 18:58:33.873068 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:33.872965 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92faba62-c6d6-4fc7-9ef2-8680f2c518eb" containerName="revoke-keys" Oct 30 18:58:33.873169 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:33.873154 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="92faba62-c6d6-4fc7-9ef2-8680f2c518eb" containerName="revoke-keys" Oct 30 18:58:33.878790 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:33.878762 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2730df4c-5q8g7" Oct 30 18:58:33.935914 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:33.935696 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:58:33.935914 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:33.935869 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:58:33.937254 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:33.937225 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:58:33.937254 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:33.937256 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:58:33.942012 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:33.941980 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2730df4c-5q8g7"] Oct 30 18:58:34.032647 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:34.032609 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f9e1a62-b37c-476a-9d84-586b373579ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2730df4c-5q8g7\" (UID: \"3f9e1a62-b37c-476a-9d84-586b373579ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2730df4c-5q8g7" Oct 30 18:58:34.032861 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:34.032809 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msqqv\" (UniqueName: \"kubernetes.io/projected/3f9e1a62-b37c-476a-9d84-586b373579ae-kube-api-access-msqqv\") pod \"maas-api-revoke-sub-2730df4c-5q8g7\" (UID: \"3f9e1a62-b37c-476a-9d84-586b373579ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2730df4c-5q8g7" Oct 30 18:58:34.032975 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:34.032961 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f9e1a62-b37c-476a-9d84-586b373579ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2730df4c-5q8g7\" (UID: \"3f9e1a62-b37c-476a-9d84-586b373579ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2730df4c-5q8g7" Oct 30 18:58:34.099562 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:34.099526 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ae1ac14-8378-4501-ba25-4e59f1e674ac" path="/var/lib/kubelet/pods/5ae1ac14-8378-4501-ba25-4e59f1e674ac/volumes" Oct 30 18:58:34.134512 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:34.134417 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msqqv\" (UniqueName: \"kubernetes.io/projected/3f9e1a62-b37c-476a-9d84-586b373579ae-kube-api-access-msqqv\") pod \"maas-api-revoke-sub-2730df4c-5q8g7\" (UID: \"3f9e1a62-b37c-476a-9d84-586b373579ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2730df4c-5q8g7" Oct 30 18:58:34.134696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:34.134522 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f9e1a62-b37c-476a-9d84-586b373579ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2730df4c-5q8g7\" (UID: \"3f9e1a62-b37c-476a-9d84-586b373579ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2730df4c-5q8g7" Oct 30 18:58:34.134696 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:34.134589 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f9e1a62-b37c-476a-9d84-586b373579ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2730df4c-5q8g7\" (UID: \"3f9e1a62-b37c-476a-9d84-586b373579ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2730df4c-5q8g7" Oct 30 18:58:34.135233 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:34.135208 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f9e1a62-b37c-476a-9d84-586b373579ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2730df4c-5q8g7\" (UID: \"3f9e1a62-b37c-476a-9d84-586b373579ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2730df4c-5q8g7" Oct 30 18:58:34.137122 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:34.137095 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f9e1a62-b37c-476a-9d84-586b373579ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2730df4c-5q8g7\" (UID: \"3f9e1a62-b37c-476a-9d84-586b373579ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2730df4c-5q8g7" Oct 30 18:58:34.195980 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:34.195945 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msqqv\" (UniqueName: \"kubernetes.io/projected/3f9e1a62-b37c-476a-9d84-586b373579ae-kube-api-access-msqqv\") pod \"maas-api-revoke-sub-2730df4c-5q8g7\" (UID: \"3f9e1a62-b37c-476a-9d84-586b373579ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2730df4c-5q8g7" Oct 30 18:58:34.494107 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:34.494077 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2730df4c-5q8g7" Oct 30 18:58:34.904501 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:34.903489 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2730df4c-5q8g7"] Oct 30 18:58:35.041343 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:35.041305 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2730df4c-5q8g7" event={"ID":"3f9e1a62-b37c-476a-9d84-586b373579ae","Type":"ContainerStarted","Data":"6c9cfc029edc4a1770ab56aa1471f6229a5fb2ef56736beb0d26c9e42fd73f58"} Oct 30 18:58:36.050064 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:36.050032 4531 generic.go:356] "Generic (PLEG): container finished" podID="3f9e1a62-b37c-476a-9d84-586b373579ae" containerID="46df501a3552f694cacce424ddbd4aef1f77f47bdfad9df5e5b417d87a559e86" exitCode=0 Oct 30 18:58:36.050500 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:36.050117 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2730df4c-5q8g7" event={"ID":"3f9e1a62-b37c-476a-9d84-586b373579ae","Type":"ContainerDied","Data":"46df501a3552f694cacce424ddbd4aef1f77f47bdfad9df5e5b417d87a559e86"} Oct 30 18:58:37.235576 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:37.235550 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2730df4c-5q8g7" Oct 30 18:58:37.388194 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:37.388070 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f9e1a62-b37c-476a-9d84-586b373579ae-kube-api-access-msqqv\" (UniqueName: \"kubernetes.io/projected/3f9e1a62-b37c-476a-9d84-586b373579ae-kube-api-access-msqqv\") pod \"3f9e1a62-b37c-476a-9d84-586b373579ae\" (UID: \"3f9e1a62-b37c-476a-9d84-586b373579ae\") " Oct 30 18:58:37.388194 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:37.388111 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f9e1a62-b37c-476a-9d84-586b373579ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f9e1a62-b37c-476a-9d84-586b373579ae-maas-api-openshift-ca\") pod \"3f9e1a62-b37c-476a-9d84-586b373579ae\" (UID: \"3f9e1a62-b37c-476a-9d84-586b373579ae\") " Oct 30 18:58:37.388463 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:37.388274 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f9e1a62-b37c-476a-9d84-586b373579ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f9e1a62-b37c-476a-9d84-586b373579ae-maas-api-xks-ca\") pod \"3f9e1a62-b37c-476a-9d84-586b373579ae\" (UID: \"3f9e1a62-b37c-476a-9d84-586b373579ae\") " Oct 30 18:58:37.388683 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:37.388649 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f9e1a62-b37c-476a-9d84-586b373579ae-maas-api-openshift-ca" pod "3f9e1a62-b37c-476a-9d84-586b373579ae" (UID: "3f9e1a62-b37c-476a-9d84-586b373579ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:37.389074 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:37.389050 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f9e1a62-b37c-476a-9d84-586b373579ae-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:58:37.391065 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:37.391029 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f9e1a62-b37c-476a-9d84-586b373579ae-maas-api-xks-ca" pod "3f9e1a62-b37c-476a-9d84-586b373579ae" (UID: "3f9e1a62-b37c-476a-9d84-586b373579ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:37.391229 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:37.391068 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9e1a62-b37c-476a-9d84-586b373579ae-kube-api-access-msqqv" pod "3f9e1a62-b37c-476a-9d84-586b373579ae" (UID: "3f9e1a62-b37c-476a-9d84-586b373579ae"). InnerVolumeSpecName "kube-api-access-msqqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:37.490630 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:37.490588 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msqqv\" (UniqueName: \"kubernetes.io/projected/3f9e1a62-b37c-476a-9d84-586b373579ae-kube-api-access-msqqv\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:58:37.490630 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:37.490623 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f9e1a62-b37c-476a-9d84-586b373579ae-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:58:38.065935 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:38.065890 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2730df4c-5q8g7" event={"ID":"3f9e1a62-b37c-476a-9d84-586b373579ae","Type":"ContainerDied","Data":"6c9cfc029edc4a1770ab56aa1471f6229a5fb2ef56736beb0d26c9e42fd73f58"} Oct 30 18:58:38.066319 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:38.066297 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2730df4c-5q8g7" Oct 30 18:58:38.068011 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:38.067850 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c9cfc029edc4a1770ab56aa1471f6229a5fb2ef56736beb0d26c9e42fd73f58" Oct 30 18:58:38.285499 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:38.285440 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k" podUID="0e569a12-fea9-4875-b068-15c7acbb0d3a" containerName="payload-processing" containerID="cri-o://b96862750592350ecad25af7037710d517badcd054558b4ed551863a2aefd225" gracePeriod=25 Oct 30 18:58:41.127521 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:41.127433 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k" podUID="0e569a12-fea9-4875-b068-15c7acbb0d3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:58:47.696033 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:47.695999 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm"] Oct 30 18:58:47.696682 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:47.696661 4531 cpu_manager.go:414] "containerMap: removing container" podUID="3f9e1a62-b37c-476a-9d84-586b373579ae" containerName="revoke-keys" Oct 30 18:58:47.696682 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:47.696681 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f9e1a62-b37c-476a-9d84-586b373579ae" containerName="revoke-keys" Oct 30 18:58:47.696907 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:47.696878 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f9e1a62-b37c-476a-9d84-586b373579ae" containerName="revoke-keys" Oct 30 18:58:47.724802 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:47.723382 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:47.784155 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:47.783224 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-132af1-openshift-default-dockercfg-blfjp\"" Oct 30 18:58:47.827234 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:47.827194 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52581cb3-251f-4450-b981-1b7fa19d9367-istio-token\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:47.852356 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:47.852311 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52581cb3-251f-4450-b981-1b7fa19d9367-istio-podinfo\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:47.867953 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:47.867916 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pkbw\" (UniqueName: \"kubernetes.io/projected/52581cb3-251f-4450-b981-1b7fa19d9367-kube-api-access-8pkbw\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:47.868313 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:47.868294 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-workload-socket\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:47.868532 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:47.868515 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-istio-envoy\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:47.880321 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:47.880284 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-workload-certs\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:47.880976 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:47.880948 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52581cb3-251f-4450-b981-1b7fa19d9367-istiod-ca-cert\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:47.898156 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:47.881233 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-credential-socket\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:47.898502 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:47.898473 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-istio-data\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:47.921393 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:47.920673 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm"] Oct 30 18:58:47.999512 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:47.999475 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-workload-socket\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:47.999725 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:47.999558 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-istio-envoy\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:47.999725 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:47.999599 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-workload-certs\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:47.999725 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:47.999660 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52581cb3-251f-4450-b981-1b7fa19d9367-istiod-ca-cert\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:47.999906 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:47.999734 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-credential-socket\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:47.999906 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:47.999791 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-istio-data\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:47.999906 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:47.999831 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52581cb3-251f-4450-b981-1b7fa19d9367-istio-token\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:48.000072 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:47.999924 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52581cb3-251f-4450-b981-1b7fa19d9367-istio-podinfo\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:48.000072 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:47.999948 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pkbw\" (UniqueName: \"kubernetes.io/projected/52581cb3-251f-4450-b981-1b7fa19d9367-kube-api-access-8pkbw\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:48.000072 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:48.000015 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-workload-socket\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:48.000379 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:48.000352 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-workload-certs\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:48.000655 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:48.000599 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-credential-socket\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:48.001075 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:48.001033 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52581cb3-251f-4450-b981-1b7fa19d9367-istiod-ca-cert\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:48.001614 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:48.001585 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-istio-data\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:48.003493 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:48.003105 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-istio-envoy\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:48.004757 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:48.004736 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52581cb3-251f-4450-b981-1b7fa19d9367-istio-podinfo\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:48.077514 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:48.077484 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pkbw\" (UniqueName: \"kubernetes.io/projected/52581cb3-251f-4450-b981-1b7fa19d9367-kube-api-access-8pkbw\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:48.099184 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:48.079996 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52581cb3-251f-4450-b981-1b7fa19d9367-istio-token\") pod \"e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:48.373755 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:48.373660 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:48.828232 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:48.828205 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm"] Oct 30 18:58:48.828587 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:58:48.828402 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52581cb3_251f_4450_b981_1b7fa19d9367.slice/crio-bd19d4a9434a5cc87beb6ff3b0ff99fe8b2b5ffe1f8cbfba993c5d91aae3e4bb WatchSource:0}: Error finding container bd19d4a9434a5cc87beb6ff3b0ff99fe8b2b5ffe1f8cbfba993c5d91aae3e4bb: Status 404 returned error can't find the container with id bd19d4a9434a5cc87beb6ff3b0ff99fe8b2b5ffe1f8cbfba993c5d91aae3e4bb Oct 30 18:58:48.831004 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:48.830964 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:58:48.831114 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:48.831039 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:58:48.831114 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:48.831070 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:58:49.072424 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:49.072383 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c985eea-b9c8k"] Oct 30 18:58:49.080437 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:49.080338 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c985eea-b9c8k"] Oct 30 18:58:49.084162 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:49.084122 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7964c56e-5rsbb"] Oct 30 18:58:49.120555 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:49.120524 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7964c56e-5rsbb"] Oct 30 18:58:49.148556 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:49.148515 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" event={"ID":"52581cb3-251f-4450-b981-1b7fa19d9367","Type":"ContainerStarted","Data":"28de017c8644ce0e679c153d9a2e123e3d8940c1bfb92cec5b413eae59930edc"} Oct 30 18:58:49.148714 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:49.148582 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" event={"ID":"52581cb3-251f-4450-b981-1b7fa19d9367","Type":"ContainerStarted","Data":"bd19d4a9434a5cc87beb6ff3b0ff99fe8b2b5ffe1f8cbfba993c5d91aae3e4bb"} Oct 30 18:58:49.359356 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:49.359244 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" podStartSLOduration=2.359227776 podStartE2EDuration="2.359227776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:49.356837316 +0000 UTC m=+1489.383942276" watchObservedRunningTime="2026-10-30 18:58:49.359227776 +0000 UTC m=+1489.386332906" Oct 30 18:58:49.373900 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:49.373874 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:49.378925 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:49.378903 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:50.102096 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:50.102059 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="670e2e17-779c-4357-b360-4eb70afb8131" path="/var/lib/kubelet/pods/670e2e17-779c-4357-b360-4eb70afb8131/volumes" Oct 30 18:58:50.102859 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:50.102833 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1" path="/var/lib/kubelet/pods/8daf35c6-a29b-4bbd-a2ea-0fb37ad163e1/volumes" Oct 30 18:58:50.156529 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:50.156495 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:50.157731 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:50.157708 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 18:58:51.127484 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:51.127442 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k" podUID="0e569a12-fea9-4875-b068-15c7acbb0d3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:58:56.441641 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:58:56.441490 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:58:56.709612 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:56.709520 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47"] Oct 30 18:58:56.721591 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:56.721556 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47" Oct 30 18:58:56.775475 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:56.775440 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-132af1\"" Oct 30 18:58:56.876786 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:56.876230 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6qst\" (UniqueName: \"kubernetes.io/projected/6f8455f6-97f8-47c7-b862-34d0a948329b-kube-api-access-j6qst\") pod \"payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47\" (UID: \"6f8455f6-97f8-47c7-b862-34d0a948329b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47" Oct 30 18:58:56.876786 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:56.876298 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f8455f6-97f8-47c7-b862-34d0a948329b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47\" (UID: \"6f8455f6-97f8-47c7-b862-34d0a948329b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47" Oct 30 18:58:56.883183 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:56.880929 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47"] Oct 30 18:58:56.980732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:56.980630 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6qst\" (UniqueName: \"kubernetes.io/projected/6f8455f6-97f8-47c7-b862-34d0a948329b-kube-api-access-j6qst\") pod \"payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47\" (UID: \"6f8455f6-97f8-47c7-b862-34d0a948329b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47" Oct 30 18:58:56.980732 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:56.980701 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f8455f6-97f8-47c7-b862-34d0a948329b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47\" (UID: \"6f8455f6-97f8-47c7-b862-34d0a948329b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47" Oct 30 18:58:56.981968 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:56.981927 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f8455f6-97f8-47c7-b862-34d0a948329b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47\" (UID: \"6f8455f6-97f8-47c7-b862-34d0a948329b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47" Oct 30 18:58:57.065349 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:57.065254 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6qst\" (UniqueName: \"kubernetes.io/projected/6f8455f6-97f8-47c7-b862-34d0a948329b-kube-api-access-j6qst\") pod \"payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47\" (UID: \"6f8455f6-97f8-47c7-b862-34d0a948329b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47" Oct 30 18:58:57.071442 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:57.071407 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47" Oct 30 18:58:57.514752 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:57.514720 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47"] Oct 30 18:58:58.080106 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:58.080071 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce18950a-dps9s"] Oct 30 18:58:58.085869 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:58.085838 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce18950a-dps9s"] Oct 30 18:58:58.099222 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:58.099191 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63efe720-284f-4533-9f8d-ebfd08dc07d1" path="/var/lib/kubelet/pods/63efe720-284f-4533-9f8d-ebfd08dc07d1/volumes" Oct 30 18:58:58.240405 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:58.239016 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47" event={"ID":"6f8455f6-97f8-47c7-b862-34d0a948329b","Type":"ContainerStarted","Data":"1d4b4dff91768fb5afc7b4176dcdf9465464203abc9df1289616113aba93eb0b"} Oct 30 18:58:58.240405 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:58.239085 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47" event={"ID":"6f8455f6-97f8-47c7-b862-34d0a948329b","Type":"ContainerStarted","Data":"4493d476598e2171f65cccec57b08632895c459064b8e8444465baab58553f63"} Oct 30 18:58:58.240405 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:58.240208 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47" Oct 30 18:58:58.356206 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:58.356077 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47" podStartSLOduration=2.356060357 podStartE2EDuration="2.356060357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:58.355099539 +0000 UTC m=+1498.382204501" watchObservedRunningTime="2026-10-30 18:58:58.356060357 +0000 UTC m=+1498.383165317" Oct 30 18:58:58.664023 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:58.663995 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k" Oct 30 18:58:58.813209 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:58.813175 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e569a12-fea9-4875-b068-15c7acbb0d3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e569a12-fea9-4875-b068-15c7acbb0d3a-plugins-config-volume\") pod \"0e569a12-fea9-4875-b068-15c7acbb0d3a\" (UID: \"0e569a12-fea9-4875-b068-15c7acbb0d3a\") " Oct 30 18:58:58.813410 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:58.813242 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e569a12-fea9-4875-b068-15c7acbb0d3a-kube-api-access-57tff\" (UniqueName: \"kubernetes.io/projected/0e569a12-fea9-4875-b068-15c7acbb0d3a-kube-api-access-57tff\") pod \"0e569a12-fea9-4875-b068-15c7acbb0d3a\" (UID: \"0e569a12-fea9-4875-b068-15c7acbb0d3a\") " Oct 30 18:58:58.813724 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:58.813699 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e569a12-fea9-4875-b068-15c7acbb0d3a-plugins-config-volume" pod "0e569a12-fea9-4875-b068-15c7acbb0d3a" (UID: "0e569a12-fea9-4875-b068-15c7acbb0d3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:58.813896 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:58.813874 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e569a12-fea9-4875-b068-15c7acbb0d3a-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:58:58.848604 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:58.848564 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e569a12-fea9-4875-b068-15c7acbb0d3a-kube-api-access-57tff" pod "0e569a12-fea9-4875-b068-15c7acbb0d3a" (UID: "0e569a12-fea9-4875-b068-15c7acbb0d3a"). InnerVolumeSpecName "kube-api-access-57tff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:58.914916 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:58.914821 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57tff\" (UniqueName: \"kubernetes.io/projected/0e569a12-fea9-4875-b068-15c7acbb0d3a-kube-api-access-57tff\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:58:59.067366 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:59.067269 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb6fa702-28sj4"] Oct 30 18:58:59.071014 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:59.070990 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb6fa702-28sj4"] Oct 30 18:58:59.250411 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:59.250365 4531 generic.go:356] "Generic (PLEG): container finished" podID="0e569a12-fea9-4875-b068-15c7acbb0d3a" containerID="b96862750592350ecad25af7037710d517badcd054558b4ed551863a2aefd225" exitCode=0 Oct 30 18:58:59.250411 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:59.250412 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k" Oct 30 18:58:59.250667 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:59.250452 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k" event={"ID":"0e569a12-fea9-4875-b068-15c7acbb0d3a","Type":"ContainerDied","Data":"b96862750592350ecad25af7037710d517badcd054558b4ed551863a2aefd225"} Oct 30 18:58:59.250667 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:59.250515 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k" event={"ID":"0e569a12-fea9-4875-b068-15c7acbb0d3a","Type":"ContainerDied","Data":"382975a118e0cba4940db4f34f2020f3dae781475f8bb997c4f8189f76eff894"} Oct 30 18:58:59.250667 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:59.250545 4531 scope.go:122] "RemoveContainer" containerID="b96862750592350ecad25af7037710d517badcd054558b4ed551863a2aefd225" Oct 30 18:58:59.267710 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:59.267688 4531 scope.go:122] "RemoveContainer" containerID="b96862750592350ecad25af7037710d517badcd054558b4ed551863a2aefd225" Oct 30 18:58:59.268039 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:58:59.268021 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b96862750592350ecad25af7037710d517badcd054558b4ed551863a2aefd225\": container with ID starting with b96862750592350ecad25af7037710d517badcd054558b4ed551863a2aefd225 not found: ID does not exist" containerID="b96862750592350ecad25af7037710d517badcd054558b4ed551863a2aefd225" Oct 30 18:58:59.268078 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:59.268049 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b96862750592350ecad25af7037710d517badcd054558b4ed551863a2aefd225"} err="failed to get container status \"b96862750592350ecad25af7037710d517badcd054558b4ed551863a2aefd225\": rpc error: code = NotFound desc = could not find container \"b96862750592350ecad25af7037710d517badcd054558b4ed551863a2aefd225\": container with ID starting with b96862750592350ecad25af7037710d517badcd054558b4ed551863a2aefd225 not found: ID does not exist" Oct 30 18:58:59.444627 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:59.444604 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k"] Oct 30 18:58:59.471396 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:58:59.471369 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-rd89k"] Oct 30 18:59:00.101080 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:00.101050 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e569a12-fea9-4875-b068-15c7acbb0d3a" path="/var/lib/kubelet/pods/0e569a12-fea9-4875-b068-15c7acbb0d3a/volumes" Oct 30 18:59:00.101572 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:00.101502 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e964bfe-cdfd-4baf-8d32-15c35847afd6" path="/var/lib/kubelet/pods/8e964bfe-cdfd-4baf-8d32-15c35847afd6/volumes" Oct 30 18:59:00.462779 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:00.462696 4531 scope.go:122] "RemoveContainer" containerID="7ade7150efeb1b3737db2bf11b06ab5037004b48bdd306ecac66bb879501c5d4" Oct 30 18:59:00.473713 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:00.473685 4531 scope.go:122] "RemoveContainer" containerID="ca4ed686aa1d4920ccaee8a2c68d015e59f98b86946f39b070e0baa4dc60935f" Oct 30 18:59:00.490547 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:00.490520 4531 scope.go:122] "RemoveContainer" containerID="4adcf418f32352900890efee8b9a9823a94aaa8b14292f9d2fb2609696c03bbb" Oct 30 18:59:03.285677 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:03.285641 4531 generic.go:356] "Generic (PLEG): container finished" podID="90016c74-97f8-4976-87c0-1603b897c2f3" containerID="595c6a03ff16516ae450f50452663d04111f911a3c4c9d0a5f4586f5f88f9c14" exitCode=0 Oct 30 18:59:03.286220 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:03.285709 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" event={"ID":"90016c74-97f8-4976-87c0-1603b897c2f3","Type":"ContainerDied","Data":"595c6a03ff16516ae450f50452663d04111f911a3c4c9d0a5f4586f5f88f9c14"} Oct 30 18:59:03.286220 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:03.286074 4531 scope.go:122] "RemoveContainer" containerID="595c6a03ff16516ae450f50452663d04111f911a3c4c9d0a5f4586f5f88f9c14" Oct 30 18:59:03.287582 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:03.287554 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:59:03.287656 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:03.287624 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:59:03.287702 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:03.287659 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:59:04.296185 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:04.296153 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" event={"ID":"90016c74-97f8-4976-87c0-1603b897c2f3","Type":"ContainerStarted","Data":"4429e7d3ec8c7564c665eeba9841957bb50b148468fcf0c15b77372bd90d04ca"} Oct 30 18:59:04.540158 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:04.534017 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:59:04.540158 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:04.534158 4531 patch_prober.go:29] interesting pod/e2e-api-b-470093-openshift-default-6745c4765-8xfqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:59:04.540158 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:04.534203 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" podUID="90016c74-97f8-4976-87c0-1603b897c2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 18:59:05.540290 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:05.540256 4531 patch_prober.go:29] interesting pod/e2e-api-b-470093-openshift-default-6745c4765-8xfqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:59:05.540813 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:05.540779 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" podUID="90016c74-97f8-4976-87c0-1603b897c2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 18:59:06.540540 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:06.540506 4531 patch_prober.go:29] interesting pod/e2e-api-b-470093-openshift-default-6745c4765-8xfqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:59:06.540961 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:06.540576 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" podUID="90016c74-97f8-4976-87c0-1603b897c2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 18:59:07.540401 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:07.540358 4531 patch_prober.go:29] interesting pod/e2e-api-b-470093-openshift-default-6745c4765-8xfqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:59:07.540593 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:07.540431 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" podUID="90016c74-97f8-4976-87c0-1603b897c2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 18:59:08.535154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:08.534266 4531 patch_prober.go:29] interesting pod/e2e-api-b-470093-openshift-default-6745c4765-8xfqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:59:08.535154 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:08.534360 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" podUID="90016c74-97f8-4976-87c0-1603b897c2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 18:59:09.526529 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:09.526493 4531 patch_prober.go:29] interesting pod/e2e-api-b-470093-openshift-default-6745c4765-8xfqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 18:59:09.526953 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:09.526546 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" podUID="90016c74-97f8-4976-87c0-1603b897c2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 18:59:09.581211 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:09.581165 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:59:10.264618 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:10.264589 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47" Oct 30 18:59:10.530803 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:10.530718 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:59:10.531858 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:10.531838 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 18:59:16.058895 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:16.058860 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a50c88dc-cwmgl"] Oct 30 18:59:16.107175 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:16.107147 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f73bd23a-hngd6"] Oct 30 18:59:16.111356 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:16.111322 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a50c88dc-cwmgl"] Oct 30 18:59:16.115100 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:16.115078 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f73bd23a-hngd6"] Oct 30 18:59:18.100206 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:18.100163 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e86dc9c-db4d-478a-8e7c-2b04900e5c04" path="/var/lib/kubelet/pods/3e86dc9c-db4d-478a-8e7c-2b04900e5c04/volumes" Oct 30 18:59:18.100691 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:18.100675 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef" path="/var/lib/kubelet/pods/d6c7cd2f-7cce-421b-8a6d-f2577b7a7bef/volumes" Oct 30 18:59:26.398466 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:59:26.398429 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:59:27.061866 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:27.061836 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1597-swvqx"] Oct 30 18:59:27.110697 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:27.110664 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cbb1597-swvqx"] Oct 30 18:59:28.099423 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:28.099386 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfb982b4-d179-4810-bbcd-e938c9797768" path="/var/lib/kubelet/pods/bfb982b4-d179-4810-bbcd-e938c9797768/volumes" Oct 30 18:59:28.099822 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:28.099715 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f947eab8-rrqtn"] Oct 30 18:59:28.141761 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:28.141716 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f947eab8-rrqtn"] Oct 30 18:59:30.102024 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:30.101986 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57e07223-5536-45a0-931f-3c3874c63a67" path="/var/lib/kubelet/pods/57e07223-5536-45a0-931f-3c3874c63a67/volumes" Oct 30 18:59:36.705078 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:36.705045 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm"] Oct 30 18:59:36.753997 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:36.753959 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-5t8lm"] Oct 30 18:59:37.121449 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:37.121414 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10236a-dd79k"] Oct 30 18:59:37.121957 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:37.121943 4531 cpu_manager.go:414] "containerMap: removing container" podUID="0e569a12-fea9-4875-b068-15c7acbb0d3a" containerName="payload-processing" Oct 30 18:59:37.122005 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:37.121959 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e569a12-fea9-4875-b068-15c7acbb0d3a" containerName="payload-processing" Oct 30 18:59:37.122086 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:37.122078 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e569a12-fea9-4875-b068-15c7acbb0d3a" containerName="payload-processing" Oct 30 18:59:37.126695 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:37.126675 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10236a-dd79k" Oct 30 18:59:37.181298 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:37.181273 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:59:37.181298 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:37.181293 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:59:37.185277 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:37.185247 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9193de57-f332-4a77-b129-71c9e196714f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-10236a-dd79k\" (UID: \"9193de57-f332-4a77-b129-71c9e196714f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10236a-dd79k" Oct 30 18:59:37.185440 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:37.185403 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t8wq\" (UniqueName: \"kubernetes.io/projected/9193de57-f332-4a77-b129-71c9e196714f-kube-api-access-2t8wq\") pod \"maas-api-revoke-keys-e2e-api-a-10236a-dd79k\" (UID: \"9193de57-f332-4a77-b129-71c9e196714f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10236a-dd79k" Oct 30 18:59:37.185440 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:37.185431 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9193de57-f332-4a77-b129-71c9e196714f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-10236a-dd79k\" (UID: \"9193de57-f332-4a77-b129-71c9e196714f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10236a-dd79k" Oct 30 18:59:37.231441 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:37.231414 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 18:59:37.231641 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:37.231619 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:59:37.234296 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:37.234278 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10236a-dd79k"] Oct 30 18:59:37.286607 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:37.286584 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t8wq\" (UniqueName: \"kubernetes.io/projected/9193de57-f332-4a77-b129-71c9e196714f-kube-api-access-2t8wq\") pod \"maas-api-revoke-keys-e2e-api-a-10236a-dd79k\" (UID: \"9193de57-f332-4a77-b129-71c9e196714f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10236a-dd79k" Oct 30 18:59:37.286742 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:37.286620 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9193de57-f332-4a77-b129-71c9e196714f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-10236a-dd79k\" (UID: \"9193de57-f332-4a77-b129-71c9e196714f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10236a-dd79k" Oct 30 18:59:37.286804 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:37.286786 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9193de57-f332-4a77-b129-71c9e196714f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-10236a-dd79k\" (UID: \"9193de57-f332-4a77-b129-71c9e196714f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10236a-dd79k" Oct 30 18:59:37.287451 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:37.287419 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9193de57-f332-4a77-b129-71c9e196714f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-10236a-dd79k\" (UID: \"9193de57-f332-4a77-b129-71c9e196714f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10236a-dd79k" Oct 30 18:59:37.289168 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:37.289146 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9193de57-f332-4a77-b129-71c9e196714f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-10236a-dd79k\" (UID: \"9193de57-f332-4a77-b129-71c9e196714f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10236a-dd79k" Oct 30 18:59:37.347965 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:37.347935 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t8wq\" (UniqueName: \"kubernetes.io/projected/9193de57-f332-4a77-b129-71c9e196714f-kube-api-access-2t8wq\") pod \"maas-api-revoke-keys-e2e-api-a-10236a-dd79k\" (UID: \"9193de57-f332-4a77-b129-71c9e196714f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10236a-dd79k" Oct 30 18:59:37.440023 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:37.439942 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10236a-dd79k" Oct 30 18:59:37.841774 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:37.841740 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10236a-dd79k"] Oct 30 18:59:37.843797 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:59:37.843769 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9193de57_f332_4a77_b129_71c9e196714f.slice/crio-cfd95301ff16b93fa226faf59af2cb783180bef3f91da40177c5104ac84fdc0e WatchSource:0}: Error finding container cfd95301ff16b93fa226faf59af2cb783180bef3f91da40177c5104ac84fdc0e: Status 404 returned error can't find the container with id cfd95301ff16b93fa226faf59af2cb783180bef3f91da40177c5104ac84fdc0e Oct 30 18:59:38.115640 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:38.113882 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92faba62-c6d6-4fc7-9ef2-8680f2c518eb" path="/var/lib/kubelet/pods/92faba62-c6d6-4fc7-9ef2-8680f2c518eb/volumes" Oct 30 18:59:38.586920 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:38.586879 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10236a-dd79k" event={"ID":"9193de57-f332-4a77-b129-71c9e196714f","Type":"ContainerStarted","Data":"fec45eca4353366acd32557d6bd821ed9b1831905b532ebe7a51dfeea04b5e52"} Oct 30 18:59:38.587179 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:38.586951 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10236a-dd79k" event={"ID":"9193de57-f332-4a77-b129-71c9e196714f","Type":"ContainerStarted","Data":"cfd95301ff16b93fa226faf59af2cb783180bef3f91da40177c5104ac84fdc0e"} Oct 30 18:59:38.714810 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:38.714754 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10236a-dd79k" podStartSLOduration=1.7147360310000002 podStartE2EDuration="1.714736031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:38.713375662 +0000 UTC m=+1538.740480602" watchObservedRunningTime="2026-10-30 18:59:38.714736031 +0000 UTC m=+1538.741840991" Oct 30 18:59:39.595641 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:39.595604 4531 generic.go:356] "Generic (PLEG): container finished" podID="9193de57-f332-4a77-b129-71c9e196714f" containerID="fec45eca4353366acd32557d6bd821ed9b1831905b532ebe7a51dfeea04b5e52" exitCode=0 Oct 30 18:59:39.596043 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:39.595692 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10236a-dd79k" event={"ID":"9193de57-f332-4a77-b129-71c9e196714f","Type":"ContainerDied","Data":"fec45eca4353366acd32557d6bd821ed9b1831905b532ebe7a51dfeea04b5e52"} Oct 30 18:59:40.751586 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:40.751564 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10236a-dd79k" Oct 30 18:59:40.839734 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:40.839689 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9193de57-f332-4a77-b129-71c9e196714f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9193de57-f332-4a77-b129-71c9e196714f-maas-api-openshift-ca\") pod \"9193de57-f332-4a77-b129-71c9e196714f\" (UID: \"9193de57-f332-4a77-b129-71c9e196714f\") " Oct 30 18:59:40.839927 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:40.839849 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9193de57-f332-4a77-b129-71c9e196714f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9193de57-f332-4a77-b129-71c9e196714f-maas-api-xks-ca\") pod \"9193de57-f332-4a77-b129-71c9e196714f\" (UID: \"9193de57-f332-4a77-b129-71c9e196714f\") " Oct 30 18:59:40.840007 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:40.839993 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9193de57-f332-4a77-b129-71c9e196714f-kube-api-access-2t8wq\" (UniqueName: \"kubernetes.io/projected/9193de57-f332-4a77-b129-71c9e196714f-kube-api-access-2t8wq\") pod \"9193de57-f332-4a77-b129-71c9e196714f\" (UID: \"9193de57-f332-4a77-b129-71c9e196714f\") " Oct 30 18:59:40.840116 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:40.840088 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9193de57-f332-4a77-b129-71c9e196714f-maas-api-openshift-ca" pod "9193de57-f332-4a77-b129-71c9e196714f" (UID: "9193de57-f332-4a77-b129-71c9e196714f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:40.840424 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:40.840397 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9193de57-f332-4a77-b129-71c9e196714f-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:59:40.842246 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:40.842215 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9193de57-f332-4a77-b129-71c9e196714f-maas-api-xks-ca" pod "9193de57-f332-4a77-b129-71c9e196714f" (UID: "9193de57-f332-4a77-b129-71c9e196714f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:40.842348 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:40.842309 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9193de57-f332-4a77-b129-71c9e196714f-kube-api-access-2t8wq" pod "9193de57-f332-4a77-b129-71c9e196714f" (UID: "9193de57-f332-4a77-b129-71c9e196714f"). InnerVolumeSpecName "kube-api-access-2t8wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:40.941342 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:40.941260 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t8wq\" (UniqueName: \"kubernetes.io/projected/9193de57-f332-4a77-b129-71c9e196714f-kube-api-access-2t8wq\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:59:40.941342 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:40.941287 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9193de57-f332-4a77-b129-71c9e196714f-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:59:41.610348 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:41.610322 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10236a-dd79k" Oct 30 18:59:41.610524 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:41.610352 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10236a-dd79k" event={"ID":"9193de57-f332-4a77-b129-71c9e196714f","Type":"ContainerDied","Data":"cfd95301ff16b93fa226faf59af2cb783180bef3f91da40177c5104ac84fdc0e"} Oct 30 18:59:41.610524 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:41.610397 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfd95301ff16b93fa226faf59af2cb783180bef3f91da40177c5104ac84fdc0e" Oct 30 18:59:43.090186 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:43.090114 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-185e5936-tvqhc"] Oct 30 18:59:43.139605 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:43.139567 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2db487af-nqs7s"] Oct 30 18:59:43.142756 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:43.142737 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-185e5936-tvqhc"] Oct 30 18:59:43.145511 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:43.145481 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2db487af-nqs7s"] Oct 30 18:59:44.100474 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:44.100434 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03b2b354-8b50-40dd-a140-354e8e36bc1a" path="/var/lib/kubelet/pods/03b2b354-8b50-40dd-a140-354e8e36bc1a/volumes" Oct 30 18:59:44.100936 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:44.100921 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4187bdc-ca1e-440c-97dd-2c9b97525a82" path="/var/lib/kubelet/pods/c4187bdc-ca1e-440c-97dd-2c9b97525a82/volumes" Oct 30 18:59:46.610229 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:46.610190 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc"] Oct 30 18:59:46.610775 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:46.610759 4531 cpu_manager.go:414] "containerMap: removing container" podUID="9193de57-f332-4a77-b129-71c9e196714f" containerName="revoke-keys" Oct 30 18:59:46.610775 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:46.610775 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9193de57-f332-4a77-b129-71c9e196714f" containerName="revoke-keys" Oct 30 18:59:46.610908 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:46.610881 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9193de57-f332-4a77-b129-71c9e196714f" containerName="revoke-keys" Oct 30 18:59:46.615701 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:46.615668 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc" Oct 30 18:59:46.671186 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:46.671144 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4ebcd2\"" Oct 30 18:59:46.680821 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:46.680213 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc"] Oct 30 18:59:46.716205 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:46.716101 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfe944db-5201-4af3-8bb1-8e616ff90b00-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc\" (UID: \"dfe944db-5201-4af3-8bb1-8e616ff90b00\") " pod="openshift-ingress/payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc" Oct 30 18:59:46.716486 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:46.716306 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6jg7\" (UniqueName: \"kubernetes.io/projected/dfe944db-5201-4af3-8bb1-8e616ff90b00-kube-api-access-h6jg7\") pod \"payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc\" (UID: \"dfe944db-5201-4af3-8bb1-8e616ff90b00\") " pod="openshift-ingress/payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc" Oct 30 18:59:46.718984 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:46.718934 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4ebcd2-dockercfg-m2k4b\"" Oct 30 18:59:46.788948 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:46.788914 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm"] Oct 30 18:59:46.794079 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:46.794052 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm" Oct 30 18:59:46.817110 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:46.817083 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6jg7\" (UniqueName: \"kubernetes.io/projected/dfe944db-5201-4af3-8bb1-8e616ff90b00-kube-api-access-h6jg7\") pod \"payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc\" (UID: \"dfe944db-5201-4af3-8bb1-8e616ff90b00\") " pod="openshift-ingress/payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc" Oct 30 18:59:46.817303 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:46.817266 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfe944db-5201-4af3-8bb1-8e616ff90b00-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc\" (UID: \"dfe944db-5201-4af3-8bb1-8e616ff90b00\") " pod="openshift-ingress/payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc" Oct 30 18:59:46.817950 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:46.817923 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfe944db-5201-4af3-8bb1-8e616ff90b00-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc\" (UID: \"dfe944db-5201-4af3-8bb1-8e616ff90b00\") " pod="openshift-ingress/payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc" Oct 30 18:59:46.853505 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:46.853472 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm"] Oct 30 18:59:46.878880 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:46.878810 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6jg7\" (UniqueName: \"kubernetes.io/projected/dfe944db-5201-4af3-8bb1-8e616ff90b00-kube-api-access-h6jg7\") pod \"payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc\" (UID: \"dfe944db-5201-4af3-8bb1-8e616ff90b00\") " pod="openshift-ingress/payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc" Oct 30 18:59:46.923938 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:46.919578 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sjt7\" (UniqueName: \"kubernetes.io/projected/c3aece38-123d-42a2-a192-298b1d6b1579-kube-api-access-9sjt7\") pod \"payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm\" (UID: \"c3aece38-123d-42a2-a192-298b1d6b1579\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm" Oct 30 18:59:46.923938 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:46.919797 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3aece38-123d-42a2-a192-298b1d6b1579-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm\" (UID: \"c3aece38-123d-42a2-a192-298b1d6b1579\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm" Oct 30 18:59:46.933963 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:46.933931 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc" Oct 30 18:59:47.021465 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:47.021429 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3aece38-123d-42a2-a192-298b1d6b1579-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm\" (UID: \"c3aece38-123d-42a2-a192-298b1d6b1579\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm" Oct 30 18:59:47.021689 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:47.021583 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sjt7\" (UniqueName: \"kubernetes.io/projected/c3aece38-123d-42a2-a192-298b1d6b1579-kube-api-access-9sjt7\") pod \"payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm\" (UID: \"c3aece38-123d-42a2-a192-298b1d6b1579\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm" Oct 30 18:59:47.022534 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:47.022492 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3aece38-123d-42a2-a192-298b1d6b1579-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm\" (UID: \"c3aece38-123d-42a2-a192-298b1d6b1579\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm" Oct 30 18:59:47.088960 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:47.088929 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sjt7\" (UniqueName: \"kubernetes.io/projected/c3aece38-123d-42a2-a192-298b1d6b1579-kube-api-access-9sjt7\") pod \"payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm\" (UID: \"c3aece38-123d-42a2-a192-298b1d6b1579\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm" Oct 30 18:59:47.109945 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:47.109922 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm" Oct 30 18:59:47.332046 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:59:47.332009 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe944db_5201_4af3_8bb1_8e616ff90b00.slice/crio-88c91e977a055c427db89be11ce1327941e8ca2c48527e7b4b0dcbac57e7570d WatchSource:0}: Error finding container 88c91e977a055c427db89be11ce1327941e8ca2c48527e7b4b0dcbac57e7570d: Status 404 returned error can't find the container with id 88c91e977a055c427db89be11ce1327941e8ca2c48527e7b4b0dcbac57e7570d Oct 30 18:59:47.338018 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:47.337993 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc"] Oct 30 18:59:47.534633 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:47.534606 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm"] Oct 30 18:59:47.665608 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:47.665569 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc" event={"ID":"dfe944db-5201-4af3-8bb1-8e616ff90b00","Type":"ContainerStarted","Data":"41c019b4d2660170d997906c1f34cbd78f55ce092484b9b69193a8cc8c5f49c2"} Oct 30 18:59:47.666152 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:47.665638 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc" event={"ID":"dfe944db-5201-4af3-8bb1-8e616ff90b00","Type":"ContainerStarted","Data":"88c91e977a055c427db89be11ce1327941e8ca2c48527e7b4b0dcbac57e7570d"} Oct 30 18:59:47.666152 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:47.665663 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc" Oct 30 18:59:47.667655 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:47.667617 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm" event={"ID":"c3aece38-123d-42a2-a192-298b1d6b1579","Type":"ContainerStarted","Data":"ef5eb488805d44a6b4fc4e568334eebf696f1319328db5e302f929bb39fc9d8e"} Oct 30 18:59:47.667808 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:47.667690 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm" Oct 30 18:59:47.667808 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:47.667709 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm" event={"ID":"c3aece38-123d-42a2-a192-298b1d6b1579","Type":"ContainerStarted","Data":"81554567542e847abfbae2f9dae0a60ab4a1fa95b5dcc2500b2c35dd00ed9f0f"} Oct 30 18:59:47.929729 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:47.929598 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc" podStartSLOduration=1.92958266 podStartE2EDuration="1.92958266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:47.805775241 +0000 UTC m=+1547.832880202" watchObservedRunningTime="2026-10-30 18:59:47.92958266 +0000 UTC m=+1547.956687678" Oct 30 18:59:47.929729 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:47.929705 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm" podStartSLOduration=1.929698597 podStartE2EDuration="1.929698597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:47.927755955 +0000 UTC m=+1547.954860917" watchObservedRunningTime="2026-10-30 18:59:47.929698597 +0000 UTC m=+1547.956803616" Oct 30 18:59:49.829346 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:49.829314 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw"] Oct 30 18:59:49.941451 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:49.941411 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv"] Oct 30 18:59:54.829873 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:54.829825 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw" podUID="7d1e0b02-28ed-4d52-b315-663f7db2d945" containerName="payload-processing" containerID="cri-o://bcb8e893c138e4c4d42baf31ac94ee7adabc1cd1aba835f79db04142e5fb2e53" gracePeriod=25 Oct 30 18:59:54.942857 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:54.942818 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv" podUID="8b970059-ff75-4aba-b1c3-cda2f2534430" containerName="payload-pre-processing" containerID="cri-o://ee29d2b09aaab897ebe1e1e91d552f424536bf4f74f5c07d5aadccdd577b5b6a" gracePeriod=25 Oct 30 18:59:55.106663 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.106637 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw" Oct 30 18:59:55.237969 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.237933 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv" Oct 30 18:59:55.248951 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.248919 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d1e0b02-28ed-4d52-b315-663f7db2d945-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d1e0b02-28ed-4d52-b315-663f7db2d945-plugins-config-volume\") pod \"7d1e0b02-28ed-4d52-b315-663f7db2d945\" (UID: \"7d1e0b02-28ed-4d52-b315-663f7db2d945\") " Oct 30 18:59:55.249246 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.249227 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d1e0b02-28ed-4d52-b315-663f7db2d945-kube-api-access-5tnrs\" (UniqueName: \"kubernetes.io/projected/7d1e0b02-28ed-4d52-b315-663f7db2d945-kube-api-access-5tnrs\") pod \"7d1e0b02-28ed-4d52-b315-663f7db2d945\" (UID: \"7d1e0b02-28ed-4d52-b315-663f7db2d945\") " Oct 30 18:59:55.249486 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.249455 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d1e0b02-28ed-4d52-b315-663f7db2d945-plugins-config-volume" pod "7d1e0b02-28ed-4d52-b315-663f7db2d945" (UID: "7d1e0b02-28ed-4d52-b315-663f7db2d945"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:55.249861 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.249832 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d1e0b02-28ed-4d52-b315-663f7db2d945-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:59:55.251805 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.251774 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d1e0b02-28ed-4d52-b315-663f7db2d945-kube-api-access-5tnrs" pod "7d1e0b02-28ed-4d52-b315-663f7db2d945" (UID: "7d1e0b02-28ed-4d52-b315-663f7db2d945"). InnerVolumeSpecName "kube-api-access-5tnrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:55.365085 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.364998 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b970059-ff75-4aba-b1c3-cda2f2534430-kube-api-access-f824x\" (UniqueName: \"kubernetes.io/projected/8b970059-ff75-4aba-b1c3-cda2f2534430-kube-api-access-f824x\") pod \"8b970059-ff75-4aba-b1c3-cda2f2534430\" (UID: \"8b970059-ff75-4aba-b1c3-cda2f2534430\") " Oct 30 18:59:55.391452 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.380581 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b970059-ff75-4aba-b1c3-cda2f2534430-kube-api-access-f824x" pod "8b970059-ff75-4aba-b1c3-cda2f2534430" (UID: "8b970059-ff75-4aba-b1c3-cda2f2534430"). InnerVolumeSpecName "kube-api-access-f824x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:55.391452 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.383827 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b970059-ff75-4aba-b1c3-cda2f2534430-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b970059-ff75-4aba-b1c3-cda2f2534430-plugins-config-volume\") pod \"8b970059-ff75-4aba-b1c3-cda2f2534430\" (UID: \"8b970059-ff75-4aba-b1c3-cda2f2534430\") " Oct 30 18:59:55.391452 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.384318 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b970059-ff75-4aba-b1c3-cda2f2534430-plugins-config-volume" pod "8b970059-ff75-4aba-b1c3-cda2f2534430" (UID: "8b970059-ff75-4aba-b1c3-cda2f2534430"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:55.391452 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.384953 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tnrs\" (UniqueName: \"kubernetes.io/projected/7d1e0b02-28ed-4d52-b315-663f7db2d945-kube-api-access-5tnrs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:59:55.391452 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.384989 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f824x\" (UniqueName: \"kubernetes.io/projected/8b970059-ff75-4aba-b1c3-cda2f2534430-kube-api-access-f824x\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:59:55.391452 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.385006 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b970059-ff75-4aba-b1c3-cda2f2534430-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 18:59:55.772909 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.772876 4531 generic.go:356] "Generic (PLEG): container finished" podID="7d1e0b02-28ed-4d52-b315-663f7db2d945" containerID="bcb8e893c138e4c4d42baf31ac94ee7adabc1cd1aba835f79db04142e5fb2e53" exitCode=0 Oct 30 18:59:55.773098 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.772925 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw" Oct 30 18:59:55.773098 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.772960 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw" event={"ID":"7d1e0b02-28ed-4d52-b315-663f7db2d945","Type":"ContainerDied","Data":"bcb8e893c138e4c4d42baf31ac94ee7adabc1cd1aba835f79db04142e5fb2e53"} Oct 30 18:59:55.773098 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.773044 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw" event={"ID":"7d1e0b02-28ed-4d52-b315-663f7db2d945","Type":"ContainerDied","Data":"a4d0e7aa57fe5bd75ef4f3ae9dd3b857c01cc279863f3390b77dcd48bc932c31"} Oct 30 18:59:55.773098 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.773068 4531 scope.go:122] "RemoveContainer" containerID="bcb8e893c138e4c4d42baf31ac94ee7adabc1cd1aba835f79db04142e5fb2e53" Oct 30 18:59:55.774552 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.774530 4531 generic.go:356] "Generic (PLEG): container finished" podID="8b970059-ff75-4aba-b1c3-cda2f2534430" containerID="ee29d2b09aaab897ebe1e1e91d552f424536bf4f74f5c07d5aadccdd577b5b6a" exitCode=0 Oct 30 18:59:55.774630 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.774576 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv" Oct 30 18:59:55.774630 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.774607 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv" event={"ID":"8b970059-ff75-4aba-b1c3-cda2f2534430","Type":"ContainerDied","Data":"ee29d2b09aaab897ebe1e1e91d552f424536bf4f74f5c07d5aadccdd577b5b6a"} Oct 30 18:59:55.774718 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.774656 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv" event={"ID":"8b970059-ff75-4aba-b1c3-cda2f2534430","Type":"ContainerDied","Data":"e950d00ebb44e0213ad4e6f3fa4456b127663fefb16fc5d6506842215ece9baa"} Oct 30 18:59:55.784281 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.784262 4531 scope.go:122] "RemoveContainer" containerID="bcb8e893c138e4c4d42baf31ac94ee7adabc1cd1aba835f79db04142e5fb2e53" Oct 30 18:59:55.784594 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:59:55.784571 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcb8e893c138e4c4d42baf31ac94ee7adabc1cd1aba835f79db04142e5fb2e53\": container with ID starting with bcb8e893c138e4c4d42baf31ac94ee7adabc1cd1aba835f79db04142e5fb2e53 not found: ID does not exist" containerID="bcb8e893c138e4c4d42baf31ac94ee7adabc1cd1aba835f79db04142e5fb2e53" Oct 30 18:59:55.784678 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.784602 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcb8e893c138e4c4d42baf31ac94ee7adabc1cd1aba835f79db04142e5fb2e53"} err="failed to get container status \"bcb8e893c138e4c4d42baf31ac94ee7adabc1cd1aba835f79db04142e5fb2e53\": rpc error: code = NotFound desc = could not find container \"bcb8e893c138e4c4d42baf31ac94ee7adabc1cd1aba835f79db04142e5fb2e53\": container with ID starting with bcb8e893c138e4c4d42baf31ac94ee7adabc1cd1aba835f79db04142e5fb2e53 not found: ID does not exist" Oct 30 18:59:55.784678 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.784620 4531 scope.go:122] "RemoveContainer" containerID="ee29d2b09aaab897ebe1e1e91d552f424536bf4f74f5c07d5aadccdd577b5b6a" Oct 30 18:59:55.794925 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.794908 4531 scope.go:122] "RemoveContainer" containerID="ee29d2b09aaab897ebe1e1e91d552f424536bf4f74f5c07d5aadccdd577b5b6a" Oct 30 18:59:55.795212 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:59:55.795197 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee29d2b09aaab897ebe1e1e91d552f424536bf4f74f5c07d5aadccdd577b5b6a\": container with ID starting with ee29d2b09aaab897ebe1e1e91d552f424536bf4f74f5c07d5aadccdd577b5b6a not found: ID does not exist" containerID="ee29d2b09aaab897ebe1e1e91d552f424536bf4f74f5c07d5aadccdd577b5b6a" Oct 30 18:59:55.795265 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.795219 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee29d2b09aaab897ebe1e1e91d552f424536bf4f74f5c07d5aadccdd577b5b6a"} err="failed to get container status \"ee29d2b09aaab897ebe1e1e91d552f424536bf4f74f5c07d5aadccdd577b5b6a\": rpc error: code = NotFound desc = could not find container \"ee29d2b09aaab897ebe1e1e91d552f424536bf4f74f5c07d5aadccdd577b5b6a\": container with ID starting with ee29d2b09aaab897ebe1e1e91d552f424536bf4f74f5c07d5aadccdd577b5b6a not found: ID does not exist" Oct 30 18:59:55.938984 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.938946 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw"] Oct 30 18:59:55.944048 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:55.944020 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-10236a-f5dbb456-nl6xw"] Oct 30 18:59:56.059676 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:56.059583 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv"] Oct 30 18:59:56.063520 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:56.063496 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-10236a-847bc7b8f9-47xjv"] Oct 30 18:59:56.100010 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:56.099979 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d1e0b02-28ed-4d52-b315-663f7db2d945" path="/var/lib/kubelet/pods/7d1e0b02-28ed-4d52-b315-663f7db2d945/volumes" Oct 30 18:59:56.100387 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:56.100373 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b970059-ff75-4aba-b1c3-cda2f2534430" path="/var/lib/kubelet/pods/8b970059-ff75-4aba-b1c3-cda2f2534430/volumes" Oct 30 18:59:56.399412 ip-10-0-133-56 kubenswrapper[4531]: E1030 18:59:56.399328 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 18:59:56.757778 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:56.757748 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t"] Oct 30 18:59:56.758370 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:56.758353 4531 cpu_manager.go:414] "containerMap: removing container" podUID="8b970059-ff75-4aba-b1c3-cda2f2534430" containerName="payload-pre-processing" Oct 30 18:59:56.758445 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:56.758372 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b970059-ff75-4aba-b1c3-cda2f2534430" containerName="payload-pre-processing" Oct 30 18:59:56.758445 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:56.758401 4531 cpu_manager.go:414] "containerMap: removing container" podUID="7d1e0b02-28ed-4d52-b315-663f7db2d945" containerName="payload-processing" Oct 30 18:59:56.758445 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:56.758406 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d1e0b02-28ed-4d52-b315-663f7db2d945" containerName="payload-processing" Oct 30 18:59:56.758544 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:56.758501 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d1e0b02-28ed-4d52-b315-663f7db2d945" containerName="payload-processing" Oct 30 18:59:56.758544 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:56.758511 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b970059-ff75-4aba-b1c3-cda2f2534430" containerName="payload-pre-processing" Oct 30 18:59:56.763275 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:56.763252 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t" Oct 30 18:59:56.816082 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:56.816055 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-66baf45c\"" Oct 30 18:59:56.819832 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:56.819807 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t"] Oct 30 18:59:56.857745 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:56.857715 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn"] Oct 30 18:59:56.862706 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:56.862681 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn" Oct 30 18:59:56.915364 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:56.912041 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40617548-d684-4609-9c1e-37b58c1e4bfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t\" (UID: \"40617548-d684-4609-9c1e-37b58c1e4bfb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t" Oct 30 18:59:56.915364 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:56.912164 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p5xk\" (UniqueName: \"kubernetes.io/projected/40617548-d684-4609-9c1e-37b58c1e4bfb-kube-api-access-9p5xk\") pod \"payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t\" (UID: \"40617548-d684-4609-9c1e-37b58c1e4bfb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t" Oct 30 18:59:56.941933 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:56.941899 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-66baf45c-dockercfg-qp6ks\"" Oct 30 18:59:56.960183 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:56.958536 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn"] Oct 30 18:59:57.013432 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:57.013316 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4rlq\" (UniqueName: \"kubernetes.io/projected/c6812606-a602-4a6d-b226-4be22c94d655-kube-api-access-b4rlq\") pod \"payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn\" (UID: \"c6812606-a602-4a6d-b226-4be22c94d655\") " pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn" Oct 30 18:59:57.013432 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:57.013411 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40617548-d684-4609-9c1e-37b58c1e4bfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t\" (UID: \"40617548-d684-4609-9c1e-37b58c1e4bfb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t" Oct 30 18:59:57.013714 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:57.013491 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6812606-a602-4a6d-b226-4be22c94d655-plugins-config-volume\") pod \"payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn\" (UID: \"c6812606-a602-4a6d-b226-4be22c94d655\") " pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn" Oct 30 18:59:57.013714 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:57.013530 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p5xk\" (UniqueName: \"kubernetes.io/projected/40617548-d684-4609-9c1e-37b58c1e4bfb-kube-api-access-9p5xk\") pod \"payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t\" (UID: \"40617548-d684-4609-9c1e-37b58c1e4bfb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t" Oct 30 18:59:57.014301 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:57.014273 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40617548-d684-4609-9c1e-37b58c1e4bfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t\" (UID: \"40617548-d684-4609-9c1e-37b58c1e4bfb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t" Oct 30 18:59:57.075695 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:57.075657 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p5xk\" (UniqueName: \"kubernetes.io/projected/40617548-d684-4609-9c1e-37b58c1e4bfb-kube-api-access-9p5xk\") pod \"payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t\" (UID: \"40617548-d684-4609-9c1e-37b58c1e4bfb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t" Oct 30 18:59:57.078336 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:57.078308 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t" Oct 30 18:59:57.115699 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:57.115653 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6812606-a602-4a6d-b226-4be22c94d655-plugins-config-volume\") pod \"payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn\" (UID: \"c6812606-a602-4a6d-b226-4be22c94d655\") " pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn" Oct 30 18:59:57.115940 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:57.115843 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4rlq\" (UniqueName: \"kubernetes.io/projected/c6812606-a602-4a6d-b226-4be22c94d655-kube-api-access-b4rlq\") pod \"payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn\" (UID: \"c6812606-a602-4a6d-b226-4be22c94d655\") " pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn" Oct 30 18:59:57.116639 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:57.116607 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6812606-a602-4a6d-b226-4be22c94d655-plugins-config-volume\") pod \"payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn\" (UID: \"c6812606-a602-4a6d-b226-4be22c94d655\") " pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn" Oct 30 18:59:57.177201 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:57.176792 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4rlq\" (UniqueName: \"kubernetes.io/projected/c6812606-a602-4a6d-b226-4be22c94d655-kube-api-access-b4rlq\") pod \"payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn\" (UID: \"c6812606-a602-4a6d-b226-4be22c94d655\") " pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn" Oct 30 18:59:57.190493 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:57.190462 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn" Oct 30 18:59:57.467810 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:57.467782 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t"] Oct 30 18:59:57.468395 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:59:57.468375 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40617548_d684_4609_9c1e_37b58c1e4bfb.slice/crio-fec6c203bfd4fd4f186f014b38fccd9fd3385ec1c176a84b983a511217e156cf WatchSource:0}: Error finding container fec6c203bfd4fd4f186f014b38fccd9fd3385ec1c176a84b983a511217e156cf: Status 404 returned error can't find the container with id fec6c203bfd4fd4f186f014b38fccd9fd3385ec1c176a84b983a511217e156cf Oct 30 18:59:57.571674 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:57.571648 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn"] Oct 30 18:59:57.572799 ip-10-0-133-56 kubenswrapper[4531]: W1030 18:59:57.572771 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6812606_a602_4a6d_b226_4be22c94d655.slice/crio-7b7002d6affd95ed8131cfbd4ad2bbf61bcf1cb72dab5e4afaacfae98cf4cd32 WatchSource:0}: Error finding container 7b7002d6affd95ed8131cfbd4ad2bbf61bcf1cb72dab5e4afaacfae98cf4cd32: Status 404 returned error can't find the container with id 7b7002d6affd95ed8131cfbd4ad2bbf61bcf1cb72dab5e4afaacfae98cf4cd32 Oct 30 18:59:57.792290 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:57.792244 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t" event={"ID":"40617548-d684-4609-9c1e-37b58c1e4bfb","Type":"ContainerStarted","Data":"234635f6e6d8cc7e59acc6812c799418644a264a39159206dc5a07d8c2fd9879"} Oct 30 18:59:57.792496 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:57.792353 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t" event={"ID":"40617548-d684-4609-9c1e-37b58c1e4bfb","Type":"ContainerStarted","Data":"fec6c203bfd4fd4f186f014b38fccd9fd3385ec1c176a84b983a511217e156cf"} Oct 30 18:59:57.792496 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:57.792389 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t" Oct 30 18:59:57.793924 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:57.793901 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn" event={"ID":"c6812606-a602-4a6d-b226-4be22c94d655","Type":"ContainerStarted","Data":"5a31c070b8016132f34144a38afb9ac313dcff0f74d3dc8dfa5ca56692c74ab7"} Oct 30 18:59:57.794020 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:57.793940 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn" event={"ID":"c6812606-a602-4a6d-b226-4be22c94d655","Type":"ContainerStarted","Data":"7b7002d6affd95ed8131cfbd4ad2bbf61bcf1cb72dab5e4afaacfae98cf4cd32"} Oct 30 18:59:57.794020 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:57.793965 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn" Oct 30 18:59:57.914415 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:57.914364 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t" podStartSLOduration=1.9143503 podStartE2EDuration="1.9143503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:57.909591689 +0000 UTC m=+1557.936696647" watchObservedRunningTime="2026-10-30 18:59:57.9143503 +0000 UTC m=+1557.941455259" Oct 30 18:59:58.037899 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:58.037830 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn" podStartSLOduration=2.037806052 podStartE2EDuration="2.037806052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:58.033436818 +0000 UTC m=+1558.060541781" watchObservedRunningTime="2026-10-30 18:59:58.037806052 +0000 UTC m=+1558.064911012" Oct 30 18:59:58.678915 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:58.678883 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm" Oct 30 18:59:58.679160 ip-10-0-133-56 kubenswrapper[4531]: I1030 18:59:58.679121 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc" Oct 30 19:00:00.175050 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.175019 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz"] Oct 30 19:00:00.179628 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.179606 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz" Oct 30 19:00:00.180360 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.180338 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72"] Oct 30 19:00:00.185532 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.185512 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc"] Oct 30 19:00:00.185632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.185586 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72" Oct 30 19:00:00.189304 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.189060 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc" Oct 30 19:00:00.225286 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.225264 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p"] Oct 30 19:00:00.229364 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.229346 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j"] Oct 30 19:00:00.229480 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.229438 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p" Oct 30 19:00:00.233828 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.233804 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:00:00.234542 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.234522 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd"] Oct 30 19:00:00.235100 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.235082 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j" Oct 30 19:00:00.240884 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.240863 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd" Oct 30 19:00:00.241045 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.241018 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6"] Oct 30 19:00:00.247677 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.247658 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6" Oct 30 19:00:00.270441 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.270417 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:00:00.270568 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.270509 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:00:00.274995 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.274970 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72"] Oct 30 19:00:00.278948 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.278923 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz"] Oct 30 19:00:00.284264 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.284237 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc"] Oct 30 19:00:00.340056 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.340003 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p"] Oct 30 19:00:00.345918 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.345876 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j"] Oct 30 19:00:00.349629 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.349598 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd"] Oct 30 19:00:00.353612 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.353572 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6"] Oct 30 19:00:00.362763 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.362728 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6n6q\" (UniqueName: \"kubernetes.io/projected/6577c714-acaf-443a-8072-0dbb83e65811-kube-api-access-t6n6q\") pod \"maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd\" (UID: \"6577c714-acaf-443a-8072-0dbb83e65811\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd" Oct 30 19:00:00.362953 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.362786 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwg8p\" (UniqueName: \"kubernetes.io/projected/3c572cf0-9276-4ecc-a756-97dc76c0ac15-kube-api-access-fwg8p\") pod \"maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz\" (UID: \"3c572cf0-9276-4ecc-a756-97dc76c0ac15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz" Oct 30 19:00:00.362953 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.362874 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0949f5d9-3405-4b6c-946e-8a707a605045-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j\" (UID: \"0949f5d9-3405-4b6c-946e-8a707a605045\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j" Oct 30 19:00:00.362953 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.362908 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9424aef8-1e44-41a3-ae4a-30d48eed876c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p\" (UID: \"9424aef8-1e44-41a3-ae4a-30d48eed876c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p" Oct 30 19:00:00.362953 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.362934 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a03d7452-d8ee-472b-a1db-70ace6d017af-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889780-jcj72\" (UID: \"a03d7452-d8ee-472b-a1db-70ace6d017af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72" Oct 30 19:00:00.363115 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.363011 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f9e740-3071-4100-ad6e-30e89d622bca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6\" (UID: \"20f9e740-3071-4100-ad6e-30e89d622bca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6" Oct 30 19:00:00.363115 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.363055 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx98q\" (UniqueName: \"kubernetes.io/projected/20f9e740-3071-4100-ad6e-30e89d622bca-kube-api-access-gx98q\") pod \"maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6\" (UID: \"20f9e740-3071-4100-ad6e-30e89d622bca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6" Oct 30 19:00:00.363232 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.363219 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v97s\" (UniqueName: \"kubernetes.io/projected/9424aef8-1e44-41a3-ae4a-30d48eed876c-kube-api-access-2v97s\") pod \"maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p\" (UID: \"9424aef8-1e44-41a3-ae4a-30d48eed876c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p" Oct 30 19:00:00.363266 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.363240 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f825e65-da5a-4cfd-8bd6-8ab39093c889-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc\" (UID: \"5f825e65-da5a-4cfd-8bd6-8ab39093c889\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc" Oct 30 19:00:00.363307 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.363269 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6577c714-acaf-443a-8072-0dbb83e65811-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd\" (UID: \"6577c714-acaf-443a-8072-0dbb83e65811\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd" Oct 30 19:00:00.363307 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.363298 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmpsn\" (UniqueName: \"kubernetes.io/projected/0949f5d9-3405-4b6c-946e-8a707a605045-kube-api-access-rmpsn\") pod \"maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j\" (UID: \"0949f5d9-3405-4b6c-946e-8a707a605045\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j" Oct 30 19:00:00.363391 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.363379 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86rnh\" (UniqueName: \"kubernetes.io/projected/a03d7452-d8ee-472b-a1db-70ace6d017af-kube-api-access-86rnh\") pod \"maas-api-key-cleanup-29889780-jcj72\" (UID: \"a03d7452-d8ee-472b-a1db-70ace6d017af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72" Oct 30 19:00:00.363430 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.363405 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c572cf0-9276-4ecc-a756-97dc76c0ac15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz\" (UID: \"3c572cf0-9276-4ecc-a756-97dc76c0ac15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz" Oct 30 19:00:00.363471 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.363432 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvsts\" (UniqueName: \"kubernetes.io/projected/5f825e65-da5a-4cfd-8bd6-8ab39093c889-kube-api-access-gvsts\") pod \"maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc\" (UID: \"5f825e65-da5a-4cfd-8bd6-8ab39093c889\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc" Oct 30 19:00:00.464956 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.464866 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86rnh\" (UniqueName: \"kubernetes.io/projected/a03d7452-d8ee-472b-a1db-70ace6d017af-kube-api-access-86rnh\") pod \"maas-api-key-cleanup-29889780-jcj72\" (UID: \"a03d7452-d8ee-472b-a1db-70ace6d017af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72" Oct 30 19:00:00.464956 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.464907 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c572cf0-9276-4ecc-a756-97dc76c0ac15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz\" (UID: \"3c572cf0-9276-4ecc-a756-97dc76c0ac15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz" Oct 30 19:00:00.464956 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.464932 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvsts\" (UniqueName: \"kubernetes.io/projected/5f825e65-da5a-4cfd-8bd6-8ab39093c889-kube-api-access-gvsts\") pod \"maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc\" (UID: \"5f825e65-da5a-4cfd-8bd6-8ab39093c889\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc" Oct 30 19:00:00.465764 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.464962 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6n6q\" (UniqueName: \"kubernetes.io/projected/6577c714-acaf-443a-8072-0dbb83e65811-kube-api-access-t6n6q\") pod \"maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd\" (UID: \"6577c714-acaf-443a-8072-0dbb83e65811\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd" Oct 30 19:00:00.465764 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.465018 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwg8p\" (UniqueName: \"kubernetes.io/projected/3c572cf0-9276-4ecc-a756-97dc76c0ac15-kube-api-access-fwg8p\") pod \"maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz\" (UID: \"3c572cf0-9276-4ecc-a756-97dc76c0ac15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz" Oct 30 19:00:00.465764 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.465061 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0949f5d9-3405-4b6c-946e-8a707a605045-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j\" (UID: \"0949f5d9-3405-4b6c-946e-8a707a605045\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j" Oct 30 19:00:00.465764 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.465486 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9424aef8-1e44-41a3-ae4a-30d48eed876c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p\" (UID: \"9424aef8-1e44-41a3-ae4a-30d48eed876c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p" Oct 30 19:00:00.465764 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.465555 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a03d7452-d8ee-472b-a1db-70ace6d017af-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889780-jcj72\" (UID: \"a03d7452-d8ee-472b-a1db-70ace6d017af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72" Oct 30 19:00:00.465764 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.465635 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f9e740-3071-4100-ad6e-30e89d622bca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6\" (UID: \"20f9e740-3071-4100-ad6e-30e89d622bca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6" Oct 30 19:00:00.465764 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.465719 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx98q\" (UniqueName: \"kubernetes.io/projected/20f9e740-3071-4100-ad6e-30e89d622bca-kube-api-access-gx98q\") pod \"maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6\" (UID: \"20f9e740-3071-4100-ad6e-30e89d622bca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6" Oct 30 19:00:00.466165 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.465844 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v97s\" (UniqueName: \"kubernetes.io/projected/9424aef8-1e44-41a3-ae4a-30d48eed876c-kube-api-access-2v97s\") pod \"maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p\" (UID: \"9424aef8-1e44-41a3-ae4a-30d48eed876c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p" Oct 30 19:00:00.466165 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.465868 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f825e65-da5a-4cfd-8bd6-8ab39093c889-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc\" (UID: \"5f825e65-da5a-4cfd-8bd6-8ab39093c889\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc" Oct 30 19:00:00.466165 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.465913 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6577c714-acaf-443a-8072-0dbb83e65811-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd\" (UID: \"6577c714-acaf-443a-8072-0dbb83e65811\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd" Oct 30 19:00:00.466165 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.465935 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0949f5d9-3405-4b6c-946e-8a707a605045-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j\" (UID: \"0949f5d9-3405-4b6c-946e-8a707a605045\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j" Oct 30 19:00:00.466165 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.465950 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmpsn\" (UniqueName: \"kubernetes.io/projected/0949f5d9-3405-4b6c-946e-8a707a605045-kube-api-access-rmpsn\") pod \"maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j\" (UID: \"0949f5d9-3405-4b6c-946e-8a707a605045\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j" Oct 30 19:00:00.466446 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.466333 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9424aef8-1e44-41a3-ae4a-30d48eed876c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p\" (UID: \"9424aef8-1e44-41a3-ae4a-30d48eed876c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p" Oct 30 19:00:00.466446 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.466366 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a03d7452-d8ee-472b-a1db-70ace6d017af-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889780-jcj72\" (UID: \"a03d7452-d8ee-472b-a1db-70ace6d017af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72" Oct 30 19:00:00.466653 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.466518 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f9e740-3071-4100-ad6e-30e89d622bca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6\" (UID: \"20f9e740-3071-4100-ad6e-30e89d622bca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6" Oct 30 19:00:00.466767 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.466732 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6577c714-acaf-443a-8072-0dbb83e65811-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd\" (UID: \"6577c714-acaf-443a-8072-0dbb83e65811\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd" Oct 30 19:00:00.467043 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.467022 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f825e65-da5a-4cfd-8bd6-8ab39093c889-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc\" (UID: \"5f825e65-da5a-4cfd-8bd6-8ab39093c889\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc" Oct 30 19:00:00.467335 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.467319 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c572cf0-9276-4ecc-a756-97dc76c0ac15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz\" (UID: \"3c572cf0-9276-4ecc-a756-97dc76c0ac15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz" Oct 30 19:00:00.526772 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.526746 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvsts\" (UniqueName: \"kubernetes.io/projected/5f825e65-da5a-4cfd-8bd6-8ab39093c889-kube-api-access-gvsts\") pod \"maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc\" (UID: \"5f825e65-da5a-4cfd-8bd6-8ab39093c889\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc" Oct 30 19:00:00.526963 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.526944 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6n6q\" (UniqueName: \"kubernetes.io/projected/6577c714-acaf-443a-8072-0dbb83e65811-kube-api-access-t6n6q\") pod \"maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd\" (UID: \"6577c714-acaf-443a-8072-0dbb83e65811\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd" Oct 30 19:00:00.531805 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.531767 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v97s\" (UniqueName: \"kubernetes.io/projected/9424aef8-1e44-41a3-ae4a-30d48eed876c-kube-api-access-2v97s\") pod \"maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p\" (UID: \"9424aef8-1e44-41a3-ae4a-30d48eed876c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p" Oct 30 19:00:00.532073 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.532054 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwg8p\" (UniqueName: \"kubernetes.io/projected/3c572cf0-9276-4ecc-a756-97dc76c0ac15-kube-api-access-fwg8p\") pod \"maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz\" (UID: \"3c572cf0-9276-4ecc-a756-97dc76c0ac15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz" Oct 30 19:00:00.532403 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.532385 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx98q\" (UniqueName: \"kubernetes.io/projected/20f9e740-3071-4100-ad6e-30e89d622bca-kube-api-access-gx98q\") pod \"maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6\" (UID: \"20f9e740-3071-4100-ad6e-30e89d622bca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6" Oct 30 19:00:00.532599 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.532576 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmpsn\" (UniqueName: \"kubernetes.io/projected/0949f5d9-3405-4b6c-946e-8a707a605045-kube-api-access-rmpsn\") pod \"maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j\" (UID: \"0949f5d9-3405-4b6c-946e-8a707a605045\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j" Oct 30 19:00:00.533325 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.533304 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86rnh\" (UniqueName: \"kubernetes.io/projected/a03d7452-d8ee-472b-a1db-70ace6d017af-kube-api-access-86rnh\") pod \"maas-api-key-cleanup-29889780-jcj72\" (UID: \"a03d7452-d8ee-472b-a1db-70ace6d017af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72" Oct 30 19:00:00.549280 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.549257 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p" Oct 30 19:00:00.551915 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.551893 4531 scope.go:122] "RemoveContainer" containerID="ce5534153763aa9d9842acdfc824b5ccfb80af20b4d96ec8021e216d3e7ae740" Oct 30 19:00:00.558617 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.558599 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j" Oct 30 19:00:00.563855 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.563836 4531 scope.go:122] "RemoveContainer" containerID="3d0ba2afa526285231168a10246f4fffb3a1461b481acfe14a662001466adf47" Oct 30 19:00:00.569033 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.569005 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6" Oct 30 19:00:00.573753 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.573734 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd" Oct 30 19:00:00.636866 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.633717 4531 scope.go:122] "RemoveContainer" containerID="92568be9d5f827ab6d7f3ea8a3fbb67bd074c3aa955399ae3e03136783ebc566" Oct 30 19:00:00.655987 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.655698 4531 scope.go:122] "RemoveContainer" containerID="dd65f2c9f304b054cfd539a3692a982516a96015b529784ea7e1f419f7be3309" Oct 30 19:00:00.684452 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.683854 4531 scope.go:122] "RemoveContainer" containerID="2535c12555bc33445094450bda76fe71c3c85368d755d4db0fbd8a34b3d00bca" Oct 30 19:00:00.709843 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.709322 4531 scope.go:122] "RemoveContainer" containerID="35765236e48f3f92d7803b5c7c09845fc344c5b4d414b7f68cffd4b79a197dfd" Oct 30 19:00:00.724233 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.724210 4531 scope.go:122] "RemoveContainer" containerID="46ea4dfe797ce69a0b8bd1942f28dc408500a47dedc45611102ae78117a3b77b" Oct 30 19:00:00.742761 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.742649 4531 scope.go:122] "RemoveContainer" containerID="9f7e62877d38381f454294d497b4433a0b02099ce0256409e5674ac1ffaad050" Oct 30 19:00:00.755935 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.755905 4531 scope.go:122] "RemoveContainer" containerID="346fd32a340c7d0daaa61f1e4727755847efede003a4bf519d6669c889babf0f" Oct 30 19:00:00.795898 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.795834 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz" Oct 30 19:00:00.803482 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.803461 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72" Oct 30 19:00:00.811046 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.810854 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc" Oct 30 19:00:00.969487 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:00.968699 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p"] Oct 30 19:00:00.974692 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:00:00.974662 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9424aef8_1e44_41a3_ae4a_30d48eed876c.slice/crio-c2eaf8eb284057d2c6c6d866d8ca5621bcc35b108d9c13cf09593bc25079de41 WatchSource:0}: Error finding container c2eaf8eb284057d2c6c6d866d8ca5621bcc35b108d9c13cf09593bc25079de41: Status 404 returned error can't find the container with id c2eaf8eb284057d2c6c6d866d8ca5621bcc35b108d9c13cf09593bc25079de41 Oct 30 19:00:01.021611 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:01.021572 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j"] Oct 30 19:00:01.032563 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:01.032525 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6"] Oct 30 19:00:01.036256 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:00:01.035518 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f9e740_3071_4100_ad6e_30e89d622bca.slice/crio-f09e4a47ce4c79e832017408c53202c8fec3d15ce5276ce8ce0b05a5f539779e WatchSource:0}: Error finding container f09e4a47ce4c79e832017408c53202c8fec3d15ce5276ce8ce0b05a5f539779e: Status 404 returned error can't find the container with id f09e4a47ce4c79e832017408c53202c8fec3d15ce5276ce8ce0b05a5f539779e Oct 30 19:00:01.202428 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:01.202400 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd"] Oct 30 19:00:01.202762 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:00:01.202606 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6577c714_acaf_443a_8072_0dbb83e65811.slice/crio-9c83e3231c6d606e879888b91d243401551efb47c3982f2804dd9cf43733f5ef WatchSource:0}: Error finding container 9c83e3231c6d606e879888b91d243401551efb47c3982f2804dd9cf43733f5ef: Status 404 returned error can't find the container with id 9c83e3231c6d606e879888b91d243401551efb47c3982f2804dd9cf43733f5ef Oct 30 19:00:01.230899 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:00:01.230863 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c572cf0_9276_4ecc_a756_97dc76c0ac15.slice/crio-40e304a59a8a7eae68ae3fb5d80dde61a39fd7bb885f8a2d1742328a6d1aadb2 WatchSource:0}: Error finding container 40e304a59a8a7eae68ae3fb5d80dde61a39fd7bb885f8a2d1742328a6d1aadb2: Status 404 returned error can't find the container with id 40e304a59a8a7eae68ae3fb5d80dde61a39fd7bb885f8a2d1742328a6d1aadb2 Oct 30 19:00:01.231262 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:01.231234 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz"] Oct 30 19:00:01.266280 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:01.266249 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72"] Oct 30 19:00:01.266790 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:00:01.266766 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda03d7452_d8ee_472b_a1db_70ace6d017af.slice/crio-4ca0a2dca44d8f68b2aa042d7c3d554eda99068b1ba237143902746e9a545738 WatchSource:0}: Error finding container 4ca0a2dca44d8f68b2aa042d7c3d554eda99068b1ba237143902746e9a545738: Status 404 returned error can't find the container with id 4ca0a2dca44d8f68b2aa042d7c3d554eda99068b1ba237143902746e9a545738 Oct 30 19:00:01.280765 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:01.280709 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc"] Oct 30 19:00:01.319429 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:00:01.319364 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f825e65_da5a_4cfd_8bd6_8ab39093c889.slice/crio-b7e3d3d10d4673c2cfa77c2690a465afa0605d1b148d66cea202e004e9349534 WatchSource:0}: Error finding container b7e3d3d10d4673c2cfa77c2690a465afa0605d1b148d66cea202e004e9349534: Status 404 returned error can't find the container with id b7e3d3d10d4673c2cfa77c2690a465afa0605d1b148d66cea202e004e9349534 Oct 30 19:00:01.841992 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:01.841947 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc" event={"ID":"5f825e65-da5a-4cfd-8bd6-8ab39093c889","Type":"ContainerStarted","Data":"b7e3d3d10d4673c2cfa77c2690a465afa0605d1b148d66cea202e004e9349534"} Oct 30 19:00:01.843417 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:01.843388 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72" event={"ID":"a03d7452-d8ee-472b-a1db-70ace6d017af","Type":"ContainerStarted","Data":"4ca0a2dca44d8f68b2aa042d7c3d554eda99068b1ba237143902746e9a545738"} Oct 30 19:00:01.845236 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:01.845201 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j" event={"ID":"0949f5d9-3405-4b6c-946e-8a707a605045","Type":"ContainerStarted","Data":"3f4009d6e40a35adc08e61f0cd7556b73f57428b1ddf6e138399a6852f9ff864"} Oct 30 19:00:01.845399 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:01.845262 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j" event={"ID":"0949f5d9-3405-4b6c-946e-8a707a605045","Type":"ContainerStarted","Data":"981550b6f59c63a675113c34729fb8343399e6482de0b0f66391e18df03195b4"} Oct 30 19:00:01.847065 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:01.847034 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p" event={"ID":"9424aef8-1e44-41a3-ae4a-30d48eed876c","Type":"ContainerStarted","Data":"ff53e046f6b5d243ae696f0b07d7b346c13d041a87618d231399327f2bdd2a01"} Oct 30 19:00:01.847242 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:01.847098 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p" event={"ID":"9424aef8-1e44-41a3-ae4a-30d48eed876c","Type":"ContainerStarted","Data":"c2eaf8eb284057d2c6c6d866d8ca5621bcc35b108d9c13cf09593bc25079de41"} Oct 30 19:00:01.848830 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:01.848793 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd" event={"ID":"6577c714-acaf-443a-8072-0dbb83e65811","Type":"ContainerStarted","Data":"c731acc7229212d14160f9421241779c953562495dee6b201f31775cf8945428"} Oct 30 19:00:01.848978 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:01.848857 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd" event={"ID":"6577c714-acaf-443a-8072-0dbb83e65811","Type":"ContainerStarted","Data":"9c83e3231c6d606e879888b91d243401551efb47c3982f2804dd9cf43733f5ef"} Oct 30 19:00:01.850596 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:01.850565 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6" event={"ID":"20f9e740-3071-4100-ad6e-30e89d622bca","Type":"ContainerStarted","Data":"50014d7714116f6b22992fc6c141ac56d05ada7b83aa3ff8a73d00d8805cce94"} Oct 30 19:00:01.850719 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:01.850624 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6" event={"ID":"20f9e740-3071-4100-ad6e-30e89d622bca","Type":"ContainerStarted","Data":"f09e4a47ce4c79e832017408c53202c8fec3d15ce5276ce8ce0b05a5f539779e"} Oct 30 19:00:01.852323 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:01.852296 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz" event={"ID":"3c572cf0-9276-4ecc-a756-97dc76c0ac15","Type":"ContainerStarted","Data":"1124088b40723864b1ecf5288406b9f08d7bffda48d7c698ce2cb4aff2df86ca"} Oct 30 19:00:01.852425 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:01.852340 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz" event={"ID":"3c572cf0-9276-4ecc-a756-97dc76c0ac15","Type":"ContainerStarted","Data":"40e304a59a8a7eae68ae3fb5d80dde61a39fd7bb885f8a2d1742328a6d1aadb2"} Oct 30 19:00:01.967584 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:01.967523 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j" podStartSLOduration=1.9675043209999998 podStartE2EDuration="1.967504321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:01.964659748 +0000 UTC m=+1561.991764708" watchObservedRunningTime="2026-10-30 19:00:01.967504321 +0000 UTC m=+1561.994609279" Oct 30 19:00:02.083534 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:02.083404 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd" podStartSLOduration=2.083381218 podStartE2EDuration="2.083381218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:02.08145559 +0000 UTC m=+1562.108560550" watchObservedRunningTime="2026-10-30 19:00:02.083381218 +0000 UTC m=+1562.110486178" Oct 30 19:00:02.199794 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:02.199710 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6" podStartSLOduration=2.199691032 podStartE2EDuration="2.199691032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:02.195913549 +0000 UTC m=+1562.223018511" watchObservedRunningTime="2026-10-30 19:00:02.199691032 +0000 UTC m=+1562.226795992" Oct 30 19:00:02.314712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:02.314625 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz" podStartSLOduration=2.314608187 podStartE2EDuration="2.314608187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:02.310044032 +0000 UTC m=+1562.337148993" watchObservedRunningTime="2026-10-30 19:00:02.314608187 +0000 UTC m=+1562.341713235" Oct 30 19:00:02.427031 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:02.426954 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p" podStartSLOduration=2.426931068 podStartE2EDuration="2.426931068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:02.425547198 +0000 UTC m=+1562.452652158" watchObservedRunningTime="2026-10-30 19:00:02.426931068 +0000 UTC m=+1562.454036028" Oct 30 19:00:02.863657 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:02.863616 4531 generic.go:356] "Generic (PLEG): container finished" podID="9424aef8-1e44-41a3-ae4a-30d48eed876c" containerID="ff53e046f6b5d243ae696f0b07d7b346c13d041a87618d231399327f2bdd2a01" exitCode=60 Oct 30 19:00:02.863834 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:02.863704 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p" event={"ID":"9424aef8-1e44-41a3-ae4a-30d48eed876c","Type":"ContainerDied","Data":"ff53e046f6b5d243ae696f0b07d7b346c13d041a87618d231399327f2bdd2a01"} Oct 30 19:00:02.864067 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:02.864048 4531 scope.go:122] "RemoveContainer" containerID="ff53e046f6b5d243ae696f0b07d7b346c13d041a87618d231399327f2bdd2a01" Oct 30 19:00:02.865881 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:02.865794 4531 generic.go:356] "Generic (PLEG): container finished" podID="6577c714-acaf-443a-8072-0dbb83e65811" containerID="c731acc7229212d14160f9421241779c953562495dee6b201f31775cf8945428" exitCode=60 Oct 30 19:00:02.866002 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:02.865880 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd" event={"ID":"6577c714-acaf-443a-8072-0dbb83e65811","Type":"ContainerDied","Data":"c731acc7229212d14160f9421241779c953562495dee6b201f31775cf8945428"} Oct 30 19:00:02.866337 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:02.866321 4531 scope.go:122] "RemoveContainer" containerID="c731acc7229212d14160f9421241779c953562495dee6b201f31775cf8945428" Oct 30 19:00:02.868209 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:02.868062 4531 generic.go:356] "Generic (PLEG): container finished" podID="20f9e740-3071-4100-ad6e-30e89d622bca" containerID="50014d7714116f6b22992fc6c141ac56d05ada7b83aa3ff8a73d00d8805cce94" exitCode=60 Oct 30 19:00:02.868297 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:02.868106 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6" event={"ID":"20f9e740-3071-4100-ad6e-30e89d622bca","Type":"ContainerDied","Data":"50014d7714116f6b22992fc6c141ac56d05ada7b83aa3ff8a73d00d8805cce94"} Oct 30 19:00:02.868643 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:02.868616 4531 scope.go:122] "RemoveContainer" containerID="50014d7714116f6b22992fc6c141ac56d05ada7b83aa3ff8a73d00d8805cce94" Oct 30 19:00:02.870806 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:02.870782 4531 generic.go:356] "Generic (PLEG): container finished" podID="3c572cf0-9276-4ecc-a756-97dc76c0ac15" containerID="1124088b40723864b1ecf5288406b9f08d7bffda48d7c698ce2cb4aff2df86ca" exitCode=60 Oct 30 19:00:02.870878 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:02.870861 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz" event={"ID":"3c572cf0-9276-4ecc-a756-97dc76c0ac15","Type":"ContainerDied","Data":"1124088b40723864b1ecf5288406b9f08d7bffda48d7c698ce2cb4aff2df86ca"} Oct 30 19:00:02.871208 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:02.871191 4531 scope.go:122] "RemoveContainer" containerID="1124088b40723864b1ecf5288406b9f08d7bffda48d7c698ce2cb4aff2df86ca" Oct 30 19:00:02.873431 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:02.873402 4531 generic.go:356] "Generic (PLEG): container finished" podID="5f825e65-da5a-4cfd-8bd6-8ab39093c889" containerID="dc12cab2f3dfd3879cf50c82ef032b4424446d59b96e137885f01e723af131d6" exitCode=60 Oct 30 19:00:02.873580 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:02.873496 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc" event={"ID":"5f825e65-da5a-4cfd-8bd6-8ab39093c889","Type":"ContainerDied","Data":"dc12cab2f3dfd3879cf50c82ef032b4424446d59b96e137885f01e723af131d6"} Oct 30 19:00:02.873819 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:02.873801 4531 scope.go:122] "RemoveContainer" containerID="dc12cab2f3dfd3879cf50c82ef032b4424446d59b96e137885f01e723af131d6" Oct 30 19:00:02.875542 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:02.875512 4531 generic.go:356] "Generic (PLEG): container finished" podID="a03d7452-d8ee-472b-a1db-70ace6d017af" containerID="3d63c6bd86fbef444a7322fdb1337057129e44c6ed544dd391a0a871683fb59d" exitCode=60 Oct 30 19:00:02.875676 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:02.875558 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72" event={"ID":"a03d7452-d8ee-472b-a1db-70ace6d017af","Type":"ContainerDied","Data":"3d63c6bd86fbef444a7322fdb1337057129e44c6ed544dd391a0a871683fb59d"} Oct 30 19:00:02.876072 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:02.875942 4531 scope.go:122] "RemoveContainer" containerID="3d63c6bd86fbef444a7322fdb1337057129e44c6ed544dd391a0a871683fb59d" Oct 30 19:00:02.877789 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:02.877751 4531 generic.go:356] "Generic (PLEG): container finished" podID="0949f5d9-3405-4b6c-946e-8a707a605045" containerID="3f4009d6e40a35adc08e61f0cd7556b73f57428b1ddf6e138399a6852f9ff864" exitCode=60 Oct 30 19:00:02.877872 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:02.877802 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j" event={"ID":"0949f5d9-3405-4b6c-946e-8a707a605045","Type":"ContainerDied","Data":"3f4009d6e40a35adc08e61f0cd7556b73f57428b1ddf6e138399a6852f9ff864"} Oct 30 19:00:02.878193 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:02.878174 4531 scope.go:122] "RemoveContainer" containerID="3f4009d6e40a35adc08e61f0cd7556b73f57428b1ddf6e138399a6852f9ff864" Oct 30 19:00:03.899998 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:03.899887 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc" event={"ID":"5f825e65-da5a-4cfd-8bd6-8ab39093c889","Type":"ContainerStarted","Data":"95b114aca468a8226fa0d02d479c05995a3375507f5a9c82eb86df7586b91f96"} Oct 30 19:00:03.902173 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:03.902123 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72" event={"ID":"a03d7452-d8ee-472b-a1db-70ace6d017af","Type":"ContainerStarted","Data":"433f11593cc1b22a18256995422ec700d2afcef99fdd4002fdd67083dd60b431"} Oct 30 19:00:03.904469 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:03.904439 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j" event={"ID":"0949f5d9-3405-4b6c-946e-8a707a605045","Type":"ContainerStarted","Data":"61e26f4fe80e8bc5ea2065b646a9beb299f33fdd12540e88e8c88dd1222ff70b"} Oct 30 19:00:03.906643 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:03.906608 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p" event={"ID":"9424aef8-1e44-41a3-ae4a-30d48eed876c","Type":"ContainerStarted","Data":"8cc58b7d4d8f775178110c9cfc5ec311ad7d6b2ad7ab948f1275d61e194aa06e"} Oct 30 19:00:03.908815 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:03.908779 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd" event={"ID":"6577c714-acaf-443a-8072-0dbb83e65811","Type":"ContainerStarted","Data":"bdd51f8f9d645277a3c9ae9630f2ca87af72800adbdeed0cbf52504a19b66927"} Oct 30 19:00:03.910932 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:03.910898 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6" event={"ID":"20f9e740-3071-4100-ad6e-30e89d622bca","Type":"ContainerStarted","Data":"4a6a9f0d226295ef0d1f6cea8e5f8b2c76984f5151755626760497f85a2f3e2c"} Oct 30 19:00:03.913027 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:03.912998 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz" event={"ID":"3c572cf0-9276-4ecc-a756-97dc76c0ac15","Type":"ContainerStarted","Data":"4711e9748381623e17eeba716172740759317d91f4f519df5c9d334ca41c711d"} Oct 30 19:00:04.018730 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.018679 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc" podStartSLOduration=4.018663488 podStartE2EDuration="4.018663488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:04.016273528 +0000 UTC m=+1564.043378489" watchObservedRunningTime="2026-10-30 19:00:04.018663488 +0000 UTC m=+1564.045768447" Oct 30 19:00:04.134925 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.134874 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72" podStartSLOduration=4.134859308 podStartE2EDuration="4.134859308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:04.132549484 +0000 UTC m=+1564.159654436" watchObservedRunningTime="2026-10-30 19:00:04.134859308 +0000 UTC m=+1564.161964266" Oct 30 19:00:04.922927 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.922832 4531 generic.go:356] "Generic (PLEG): container finished" podID="5f825e65-da5a-4cfd-8bd6-8ab39093c889" containerID="95b114aca468a8226fa0d02d479c05995a3375507f5a9c82eb86df7586b91f96" exitCode=60 Oct 30 19:00:04.922927 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.922881 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc" event={"ID":"5f825e65-da5a-4cfd-8bd6-8ab39093c889","Type":"ContainerDied","Data":"95b114aca468a8226fa0d02d479c05995a3375507f5a9c82eb86df7586b91f96"} Oct 30 19:00:04.923472 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.922954 4531 scope.go:122] "RemoveContainer" containerID="dc12cab2f3dfd3879cf50c82ef032b4424446d59b96e137885f01e723af131d6" Oct 30 19:00:04.923472 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.923308 4531 scope.go:122] "RemoveContainer" containerID="95b114aca468a8226fa0d02d479c05995a3375507f5a9c82eb86df7586b91f96" Oct 30 19:00:04.923690 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:00:04.923668 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc_odh-ai-gateway-infra(5f825e65-da5a-4cfd-8bd6-8ab39093c889)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc" podUID="5f825e65-da5a-4cfd-8bd6-8ab39093c889" Oct 30 19:00:04.925025 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.925002 4531 generic.go:356] "Generic (PLEG): container finished" podID="a03d7452-d8ee-472b-a1db-70ace6d017af" containerID="433f11593cc1b22a18256995422ec700d2afcef99fdd4002fdd67083dd60b431" exitCode=60 Oct 30 19:00:04.925164 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.925075 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72" event={"ID":"a03d7452-d8ee-472b-a1db-70ace6d017af","Type":"ContainerDied","Data":"433f11593cc1b22a18256995422ec700d2afcef99fdd4002fdd67083dd60b431"} Oct 30 19:00:04.925528 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.925510 4531 scope.go:122] "RemoveContainer" containerID="433f11593cc1b22a18256995422ec700d2afcef99fdd4002fdd67083dd60b431" Oct 30 19:00:04.925850 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:00:04.925827 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889780-jcj72_odh-ai-gateway-infra(a03d7452-d8ee-472b-a1db-70ace6d017af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72" podUID="a03d7452-d8ee-472b-a1db-70ace6d017af" Oct 30 19:00:04.927016 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.926985 4531 generic.go:356] "Generic (PLEG): container finished" podID="0949f5d9-3405-4b6c-946e-8a707a605045" containerID="61e26f4fe80e8bc5ea2065b646a9beb299f33fdd12540e88e8c88dd1222ff70b" exitCode=60 Oct 30 19:00:04.927105 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.927063 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j" event={"ID":"0949f5d9-3405-4b6c-946e-8a707a605045","Type":"ContainerDied","Data":"61e26f4fe80e8bc5ea2065b646a9beb299f33fdd12540e88e8c88dd1222ff70b"} Oct 30 19:00:04.927317 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.927298 4531 scope.go:122] "RemoveContainer" containerID="61e26f4fe80e8bc5ea2065b646a9beb299f33fdd12540e88e8c88dd1222ff70b" Oct 30 19:00:04.927603 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:00:04.927578 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j_odh-ai-gateway-infra(0949f5d9-3405-4b6c-946e-8a707a605045)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j" podUID="0949f5d9-3405-4b6c-946e-8a707a605045" Oct 30 19:00:04.929087 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.929064 4531 generic.go:356] "Generic (PLEG): container finished" podID="9424aef8-1e44-41a3-ae4a-30d48eed876c" containerID="8cc58b7d4d8f775178110c9cfc5ec311ad7d6b2ad7ab948f1275d61e194aa06e" exitCode=60 Oct 30 19:00:04.929217 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.929163 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p" event={"ID":"9424aef8-1e44-41a3-ae4a-30d48eed876c","Type":"ContainerDied","Data":"8cc58b7d4d8f775178110c9cfc5ec311ad7d6b2ad7ab948f1275d61e194aa06e"} Oct 30 19:00:04.929389 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.929364 4531 scope.go:122] "RemoveContainer" containerID="8cc58b7d4d8f775178110c9cfc5ec311ad7d6b2ad7ab948f1275d61e194aa06e" Oct 30 19:00:04.929685 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:00:04.929662 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p_odh-ai-gateway-infra(9424aef8-1e44-41a3-ae4a-30d48eed876c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p" podUID="9424aef8-1e44-41a3-ae4a-30d48eed876c" Oct 30 19:00:04.931114 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.931095 4531 generic.go:356] "Generic (PLEG): container finished" podID="6577c714-acaf-443a-8072-0dbb83e65811" containerID="bdd51f8f9d645277a3c9ae9630f2ca87af72800adbdeed0cbf52504a19b66927" exitCode=60 Oct 30 19:00:04.931222 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.931160 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd" event={"ID":"6577c714-acaf-443a-8072-0dbb83e65811","Type":"ContainerDied","Data":"bdd51f8f9d645277a3c9ae9630f2ca87af72800adbdeed0cbf52504a19b66927"} Oct 30 19:00:04.931435 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.931419 4531 scope.go:122] "RemoveContainer" containerID="bdd51f8f9d645277a3c9ae9630f2ca87af72800adbdeed0cbf52504a19b66927" Oct 30 19:00:04.931717 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:00:04.931697 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd_odh-ai-gateway-infra(6577c714-acaf-443a-8072-0dbb83e65811)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd" podUID="6577c714-acaf-443a-8072-0dbb83e65811" Oct 30 19:00:04.933044 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.933026 4531 generic.go:356] "Generic (PLEG): container finished" podID="20f9e740-3071-4100-ad6e-30e89d622bca" containerID="4a6a9f0d226295ef0d1f6cea8e5f8b2c76984f5151755626760497f85a2f3e2c" exitCode=60 Oct 30 19:00:04.933235 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.933106 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6" event={"ID":"20f9e740-3071-4100-ad6e-30e89d622bca","Type":"ContainerDied","Data":"4a6a9f0d226295ef0d1f6cea8e5f8b2c76984f5151755626760497f85a2f3e2c"} Oct 30 19:00:04.933339 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.933320 4531 scope.go:122] "RemoveContainer" containerID="4a6a9f0d226295ef0d1f6cea8e5f8b2c76984f5151755626760497f85a2f3e2c" Oct 30 19:00:04.933599 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:00:04.933580 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6_odh-ai-gateway-infra(20f9e740-3071-4100-ad6e-30e89d622bca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6" podUID="20f9e740-3071-4100-ad6e-30e89d622bca" Oct 30 19:00:04.934986 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.934966 4531 generic.go:356] "Generic (PLEG): container finished" podID="3c572cf0-9276-4ecc-a756-97dc76c0ac15" containerID="4711e9748381623e17eeba716172740759317d91f4f519df5c9d334ca41c711d" exitCode=60 Oct 30 19:00:04.935059 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.935045 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz" event={"ID":"3c572cf0-9276-4ecc-a756-97dc76c0ac15","Type":"ContainerDied","Data":"4711e9748381623e17eeba716172740759317d91f4f519df5c9d334ca41c711d"} Oct 30 19:00:04.935229 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.935215 4531 scope.go:122] "RemoveContainer" containerID="4711e9748381623e17eeba716172740759317d91f4f519df5c9d334ca41c711d" Oct 30 19:00:04.935415 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:00:04.935399 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz_odh-ai-gateway-infra(3c572cf0-9276-4ecc-a756-97dc76c0ac15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz" podUID="3c572cf0-9276-4ecc-a756-97dc76c0ac15" Oct 30 19:00:04.937934 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.937916 4531 scope.go:122] "RemoveContainer" containerID="3d63c6bd86fbef444a7322fdb1337057129e44c6ed544dd391a0a871683fb59d" Oct 30 19:00:04.951188 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.951171 4531 scope.go:122] "RemoveContainer" containerID="3f4009d6e40a35adc08e61f0cd7556b73f57428b1ddf6e138399a6852f9ff864" Oct 30 19:00:04.964653 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.964631 4531 scope.go:122] "RemoveContainer" containerID="ff53e046f6b5d243ae696f0b07d7b346c13d041a87618d231399327f2bdd2a01" Oct 30 19:00:04.979230 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.979210 4531 scope.go:122] "RemoveContainer" containerID="c731acc7229212d14160f9421241779c953562495dee6b201f31775cf8945428" Oct 30 19:00:04.993918 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:04.993893 4531 scope.go:122] "RemoveContainer" containerID="50014d7714116f6b22992fc6c141ac56d05ada7b83aa3ff8a73d00d8805cce94" Oct 30 19:00:05.010315 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:05.010289 4531 scope.go:122] "RemoveContainer" containerID="1124088b40723864b1ecf5288406b9f08d7bffda48d7c698ce2cb4aff2df86ca" Oct 30 19:00:05.951305 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:05.951275 4531 scope.go:122] "RemoveContainer" containerID="95b114aca468a8226fa0d02d479c05995a3375507f5a9c82eb86df7586b91f96" Oct 30 19:00:05.951786 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:00:05.951606 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc_odh-ai-gateway-infra(5f825e65-da5a-4cfd-8bd6-8ab39093c889)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc" podUID="5f825e65-da5a-4cfd-8bd6-8ab39093c889" Oct 30 19:00:05.953205 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:05.953184 4531 scope.go:122] "RemoveContainer" containerID="433f11593cc1b22a18256995422ec700d2afcef99fdd4002fdd67083dd60b431" Oct 30 19:00:05.953470 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:00:05.953449 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889780-jcj72_odh-ai-gateway-infra(a03d7452-d8ee-472b-a1db-70ace6d017af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72" podUID="a03d7452-d8ee-472b-a1db-70ace6d017af" Oct 30 19:00:05.954634 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:05.954611 4531 scope.go:122] "RemoveContainer" containerID="61e26f4fe80e8bc5ea2065b646a9beb299f33fdd12540e88e8c88dd1222ff70b" Oct 30 19:00:05.954837 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:00:05.954817 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j_odh-ai-gateway-infra(0949f5d9-3405-4b6c-946e-8a707a605045)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j" podUID="0949f5d9-3405-4b6c-946e-8a707a605045" Oct 30 19:00:05.956066 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:05.956050 4531 scope.go:122] "RemoveContainer" containerID="8cc58b7d4d8f775178110c9cfc5ec311ad7d6b2ad7ab948f1275d61e194aa06e" Oct 30 19:00:05.956286 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:00:05.956259 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p_odh-ai-gateway-infra(9424aef8-1e44-41a3-ae4a-30d48eed876c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p" podUID="9424aef8-1e44-41a3-ae4a-30d48eed876c" Oct 30 19:00:05.957541 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:05.957525 4531 scope.go:122] "RemoveContainer" containerID="bdd51f8f9d645277a3c9ae9630f2ca87af72800adbdeed0cbf52504a19b66927" Oct 30 19:00:05.957745 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:00:05.957728 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd_odh-ai-gateway-infra(6577c714-acaf-443a-8072-0dbb83e65811)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd" podUID="6577c714-acaf-443a-8072-0dbb83e65811" Oct 30 19:00:05.958989 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:05.958974 4531 scope.go:122] "RemoveContainer" containerID="4a6a9f0d226295ef0d1f6cea8e5f8b2c76984f5151755626760497f85a2f3e2c" Oct 30 19:00:05.959242 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:00:05.959222 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6_odh-ai-gateway-infra(20f9e740-3071-4100-ad6e-30e89d622bca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6" podUID="20f9e740-3071-4100-ad6e-30e89d622bca" Oct 30 19:00:05.960423 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:05.960408 4531 scope.go:122] "RemoveContainer" containerID="4711e9748381623e17eeba716172740759317d91f4f519df5c9d334ca41c711d" Oct 30 19:00:05.960581 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:00:05.960566 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz_odh-ai-gateway-infra(3c572cf0-9276-4ecc-a756-97dc76c0ac15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz" podUID="3c572cf0-9276-4ecc-a756-97dc76c0ac15" Oct 30 19:00:08.806437 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:08.806406 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t" Oct 30 19:00:08.806942 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:08.806475 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn" Oct 30 19:00:15.092870 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:15.092784 4531 scope.go:122] "RemoveContainer" containerID="4a6a9f0d226295ef0d1f6cea8e5f8b2c76984f5151755626760497f85a2f3e2c" Oct 30 19:00:16.054768 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:16.054731 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6" event={"ID":"20f9e740-3071-4100-ad6e-30e89d622bca","Type":"ContainerStarted","Data":"7b96a18edda2214f80ff74f48dd6b0c65a598148bf6cbe0ef9a458fad0d4f2a0"} Oct 30 19:00:16.092197 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:16.092162 4531 scope.go:122] "RemoveContainer" containerID="bdd51f8f9d645277a3c9ae9630f2ca87af72800adbdeed0cbf52504a19b66927" Oct 30 19:00:16.092392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:16.092250 4531 scope.go:122] "RemoveContainer" containerID="4711e9748381623e17eeba716172740759317d91f4f519df5c9d334ca41c711d" Oct 30 19:00:16.092392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:16.092353 4531 scope.go:122] "RemoveContainer" containerID="95b114aca468a8226fa0d02d479c05995a3375507f5a9c82eb86df7586b91f96" Oct 30 19:00:17.065513 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:17.065474 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz" event={"ID":"3c572cf0-9276-4ecc-a756-97dc76c0ac15","Type":"ContainerStarted","Data":"801b2b9c735b2da8b4bc8f72329c8e4759220747f9bb955e68485559948ec181"} Oct 30 19:00:17.067464 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:17.067436 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc" event={"ID":"5f825e65-da5a-4cfd-8bd6-8ab39093c889","Type":"ContainerStarted","Data":"bb21c95ad14104211d484425c0355829eed6a83e93585c2c97bd0f83ea6d2d8a"} Oct 30 19:00:17.069191 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:17.069170 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd" event={"ID":"6577c714-acaf-443a-8072-0dbb83e65811","Type":"ContainerStarted","Data":"95d1b63089e97985391c2ea061300e70c2a4261cf508fd984a7699298d36f049"} Oct 30 19:00:17.070884 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:17.070861 4531 generic.go:356] "Generic (PLEG): container finished" podID="20f9e740-3071-4100-ad6e-30e89d622bca" containerID="7b96a18edda2214f80ff74f48dd6b0c65a598148bf6cbe0ef9a458fad0d4f2a0" exitCode=60 Oct 30 19:00:17.070980 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:17.070927 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6" event={"ID":"20f9e740-3071-4100-ad6e-30e89d622bca","Type":"ContainerDied","Data":"7b96a18edda2214f80ff74f48dd6b0c65a598148bf6cbe0ef9a458fad0d4f2a0"} Oct 30 19:00:17.070980 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:17.070974 4531 scope.go:122] "RemoveContainer" containerID="4a6a9f0d226295ef0d1f6cea8e5f8b2c76984f5151755626760497f85a2f3e2c" Oct 30 19:00:17.071293 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:17.071274 4531 scope.go:122] "RemoveContainer" containerID="7b96a18edda2214f80ff74f48dd6b0c65a598148bf6cbe0ef9a458fad0d4f2a0" Oct 30 19:00:17.071534 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:00:17.071514 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6_odh-ai-gateway-infra(20f9e740-3071-4100-ad6e-30e89d622bca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6" podUID="20f9e740-3071-4100-ad6e-30e89d622bca" Oct 30 19:00:17.092558 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:17.092540 4531 scope.go:122] "RemoveContainer" containerID="61e26f4fe80e8bc5ea2065b646a9beb299f33fdd12540e88e8c88dd1222ff70b" Oct 30 19:00:17.236565 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:17.234245 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6"] Oct 30 19:00:18.081080 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:18.081046 4531 generic.go:356] "Generic (PLEG): container finished" podID="3c572cf0-9276-4ecc-a756-97dc76c0ac15" containerID="801b2b9c735b2da8b4bc8f72329c8e4759220747f9bb955e68485559948ec181" exitCode=60 Oct 30 19:00:18.081651 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:18.081158 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz" event={"ID":"3c572cf0-9276-4ecc-a756-97dc76c0ac15","Type":"ContainerDied","Data":"801b2b9c735b2da8b4bc8f72329c8e4759220747f9bb955e68485559948ec181"} Oct 30 19:00:18.081651 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:18.081251 4531 scope.go:122] "RemoveContainer" containerID="4711e9748381623e17eeba716172740759317d91f4f519df5c9d334ca41c711d" Oct 30 19:00:18.081775 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:18.081745 4531 scope.go:122] "RemoveContainer" containerID="801b2b9c735b2da8b4bc8f72329c8e4759220747f9bb955e68485559948ec181" Oct 30 19:00:18.082378 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:00:18.082230 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz_odh-ai-gateway-infra(3c572cf0-9276-4ecc-a756-97dc76c0ac15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz" podUID="3c572cf0-9276-4ecc-a756-97dc76c0ac15" Oct 30 19:00:18.083478 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:18.083449 4531 generic.go:356] "Generic (PLEG): container finished" podID="5f825e65-da5a-4cfd-8bd6-8ab39093c889" containerID="bb21c95ad14104211d484425c0355829eed6a83e93585c2c97bd0f83ea6d2d8a" exitCode=60 Oct 30 19:00:18.083578 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:18.083520 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc" event={"ID":"5f825e65-da5a-4cfd-8bd6-8ab39093c889","Type":"ContainerDied","Data":"bb21c95ad14104211d484425c0355829eed6a83e93585c2c97bd0f83ea6d2d8a"} Oct 30 19:00:18.083997 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:18.083978 4531 scope.go:122] "RemoveContainer" containerID="bb21c95ad14104211d484425c0355829eed6a83e93585c2c97bd0f83ea6d2d8a" Oct 30 19:00:18.084376 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:00:18.084355 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc_odh-ai-gateway-infra(5f825e65-da5a-4cfd-8bd6-8ab39093c889)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc" podUID="5f825e65-da5a-4cfd-8bd6-8ab39093c889" Oct 30 19:00:18.085762 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:18.085738 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j" event={"ID":"0949f5d9-3405-4b6c-946e-8a707a605045","Type":"ContainerStarted","Data":"8769ecc1e6742f94078f1fa731c6779a7293165350c031d2657433486dbb1054"} Oct 30 19:00:18.088255 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:18.088204 4531 generic.go:356] "Generic (PLEG): container finished" podID="6577c714-acaf-443a-8072-0dbb83e65811" containerID="95d1b63089e97985391c2ea061300e70c2a4261cf508fd984a7699298d36f049" exitCode=60 Oct 30 19:00:18.088393 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:18.088284 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd" event={"ID":"6577c714-acaf-443a-8072-0dbb83e65811","Type":"ContainerDied","Data":"95d1b63089e97985391c2ea061300e70c2a4261cf508fd984a7699298d36f049"} Oct 30 19:00:18.089057 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:18.089034 4531 scope.go:122] "RemoveContainer" containerID="95d1b63089e97985391c2ea061300e70c2a4261cf508fd984a7699298d36f049" Oct 30 19:00:18.089489 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:00:18.089437 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd_odh-ai-gateway-infra(6577c714-acaf-443a-8072-0dbb83e65811)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd" podUID="6577c714-acaf-443a-8072-0dbb83e65811" Oct 30 19:00:18.092385 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:18.092222 4531 scope.go:122] "RemoveContainer" containerID="433f11593cc1b22a18256995422ec700d2afcef99fdd4002fdd67083dd60b431" Oct 30 19:00:18.093222 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:18.093205 4531 scope.go:122] "RemoveContainer" containerID="8cc58b7d4d8f775178110c9cfc5ec311ad7d6b2ad7ab948f1275d61e194aa06e" Oct 30 19:00:18.101552 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:18.101524 4531 scope.go:122] "RemoveContainer" containerID="95b114aca468a8226fa0d02d479c05995a3375507f5a9c82eb86df7586b91f96" Oct 30 19:00:18.158602 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:18.158325 4531 scope.go:122] "RemoveContainer" containerID="bdd51f8f9d645277a3c9ae9630f2ca87af72800adbdeed0cbf52504a19b66927" Oct 30 19:00:18.213350 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:18.213293 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz"] Oct 30 19:00:18.376590 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:18.376555 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6" Oct 30 19:00:18.380293 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:18.380257 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd"] Oct 30 19:00:18.441671 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:18.432005 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20f9e740-3071-4100-ad6e-30e89d622bca-kube-api-access-gx98q\" (UniqueName: \"kubernetes.io/projected/20f9e740-3071-4100-ad6e-30e89d622bca-kube-api-access-gx98q\") pod \"20f9e740-3071-4100-ad6e-30e89d622bca\" (UID: \"20f9e740-3071-4100-ad6e-30e89d622bca\") " Oct 30 19:00:18.441671 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:18.432071 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20f9e740-3071-4100-ad6e-30e89d622bca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f9e740-3071-4100-ad6e-30e89d622bca-maas-api-service-ca\") pod \"20f9e740-3071-4100-ad6e-30e89d622bca\" (UID: \"20f9e740-3071-4100-ad6e-30e89d622bca\") " Oct 30 19:00:18.456307 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:18.451380 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20f9e740-3071-4100-ad6e-30e89d622bca-maas-api-service-ca" pod "20f9e740-3071-4100-ad6e-30e89d622bca" (UID: "20f9e740-3071-4100-ad6e-30e89d622bca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:18.457354 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:18.457098 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f9e740-3071-4100-ad6e-30e89d622bca-kube-api-access-gx98q" pod "20f9e740-3071-4100-ad6e-30e89d622bca" (UID: "20f9e740-3071-4100-ad6e-30e89d622bca"). InnerVolumeSpecName "kube-api-access-gx98q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:18.554159 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:18.545357 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx98q\" (UniqueName: \"kubernetes.io/projected/20f9e740-3071-4100-ad6e-30e89d622bca-kube-api-access-gx98q\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:00:18.554159 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:18.545426 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f9e740-3071-4100-ad6e-30e89d622bca-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:00:18.648177 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:18.646212 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc"] Oct 30 19:00:19.102355 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.102319 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72" event={"ID":"a03d7452-d8ee-472b-a1db-70ace6d017af","Type":"ContainerStarted","Data":"e7ecec941ae1cd840500e95c31962a94e6158f59a45960b7ab638415680a1816"} Oct 30 19:00:19.105194 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.105159 4531 generic.go:356] "Generic (PLEG): container finished" podID="0949f5d9-3405-4b6c-946e-8a707a605045" containerID="8769ecc1e6742f94078f1fa731c6779a7293165350c031d2657433486dbb1054" exitCode=60 Oct 30 19:00:19.105386 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.105163 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j" event={"ID":"0949f5d9-3405-4b6c-946e-8a707a605045","Type":"ContainerDied","Data":"8769ecc1e6742f94078f1fa731c6779a7293165350c031d2657433486dbb1054"} Oct 30 19:00:19.105386 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.105313 4531 scope.go:122] "RemoveContainer" containerID="61e26f4fe80e8bc5ea2065b646a9beb299f33fdd12540e88e8c88dd1222ff70b" Oct 30 19:00:19.105856 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.105831 4531 scope.go:122] "RemoveContainer" containerID="8769ecc1e6742f94078f1fa731c6779a7293165350c031d2657433486dbb1054" Oct 30 19:00:19.106265 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:00:19.106237 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j_odh-ai-gateway-infra(0949f5d9-3405-4b6c-946e-8a707a605045)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j" podUID="0949f5d9-3405-4b6c-946e-8a707a605045" Oct 30 19:00:19.108506 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.108482 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p" event={"ID":"9424aef8-1e44-41a3-ae4a-30d48eed876c","Type":"ContainerStarted","Data":"d3d26eeb0796ca5ec8d0bfc03a2b4379383a366757d88be06ea6125a9162f173"} Oct 30 19:00:19.112388 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.112360 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6" Oct 30 19:00:19.112388 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.112376 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6" event={"ID":"20f9e740-3071-4100-ad6e-30e89d622bca","Type":"ContainerDied","Data":"f09e4a47ce4c79e832017408c53202c8fec3d15ce5276ce8ce0b05a5f539779e"} Oct 30 19:00:19.149896 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.149851 4531 scope.go:122] "RemoveContainer" containerID="7b96a18edda2214f80ff74f48dd6b0c65a598148bf6cbe0ef9a458fad0d4f2a0" Oct 30 19:00:19.403621 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.401317 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j"] Oct 30 19:00:19.504262 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.504214 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc" Oct 30 19:00:19.510356 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.510323 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd" Oct 30 19:00:19.577077 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.577043 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz" Oct 30 19:00:19.597020 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.596970 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f825e65-da5a-4cfd-8bd6-8ab39093c889-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f825e65-da5a-4cfd-8bd6-8ab39093c889-maas-api-service-ca\") pod \"5f825e65-da5a-4cfd-8bd6-8ab39093c889\" (UID: \"5f825e65-da5a-4cfd-8bd6-8ab39093c889\") " Oct 30 19:00:19.597233 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.597119 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f825e65-da5a-4cfd-8bd6-8ab39093c889-kube-api-access-gvsts\" (UniqueName: \"kubernetes.io/projected/5f825e65-da5a-4cfd-8bd6-8ab39093c889-kube-api-access-gvsts\") pod \"5f825e65-da5a-4cfd-8bd6-8ab39093c889\" (UID: \"5f825e65-da5a-4cfd-8bd6-8ab39093c889\") " Oct 30 19:00:19.597481 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.597455 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f825e65-da5a-4cfd-8bd6-8ab39093c889-maas-api-service-ca" pod "5f825e65-da5a-4cfd-8bd6-8ab39093c889" (UID: "5f825e65-da5a-4cfd-8bd6-8ab39093c889"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:19.597737 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.597712 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f825e65-da5a-4cfd-8bd6-8ab39093c889-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:00:19.599589 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.599555 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f825e65-da5a-4cfd-8bd6-8ab39093c889-kube-api-access-gvsts" pod "5f825e65-da5a-4cfd-8bd6-8ab39093c889" (UID: "5f825e65-da5a-4cfd-8bd6-8ab39093c889"). InnerVolumeSpecName "kube-api-access-gvsts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:19.698699 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.698607 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6577c714-acaf-443a-8072-0dbb83e65811-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6577c714-acaf-443a-8072-0dbb83e65811-maas-api-service-ca\") pod \"6577c714-acaf-443a-8072-0dbb83e65811\" (UID: \"6577c714-acaf-443a-8072-0dbb83e65811\") " Oct 30 19:00:19.698866 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.698802 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c572cf0-9276-4ecc-a756-97dc76c0ac15-kube-api-access-fwg8p\" (UniqueName: \"kubernetes.io/projected/3c572cf0-9276-4ecc-a756-97dc76c0ac15-kube-api-access-fwg8p\") pod \"3c572cf0-9276-4ecc-a756-97dc76c0ac15\" (UID: \"3c572cf0-9276-4ecc-a756-97dc76c0ac15\") " Oct 30 19:00:19.698940 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.698873 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6577c714-acaf-443a-8072-0dbb83e65811-kube-api-access-t6n6q\" (UniqueName: \"kubernetes.io/projected/6577c714-acaf-443a-8072-0dbb83e65811-kube-api-access-t6n6q\") pod \"6577c714-acaf-443a-8072-0dbb83e65811\" (UID: \"6577c714-acaf-443a-8072-0dbb83e65811\") " Oct 30 19:00:19.698995 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.698941 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c572cf0-9276-4ecc-a756-97dc76c0ac15-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c572cf0-9276-4ecc-a756-97dc76c0ac15-maas-api-service-ca\") pod \"3c572cf0-9276-4ecc-a756-97dc76c0ac15\" (UID: \"3c572cf0-9276-4ecc-a756-97dc76c0ac15\") " Oct 30 19:00:19.699074 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.699050 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6577c714-acaf-443a-8072-0dbb83e65811-maas-api-service-ca" pod "6577c714-acaf-443a-8072-0dbb83e65811" (UID: "6577c714-acaf-443a-8072-0dbb83e65811"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:19.699363 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.699331 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c572cf0-9276-4ecc-a756-97dc76c0ac15-maas-api-service-ca" pod "3c572cf0-9276-4ecc-a756-97dc76c0ac15" (UID: "3c572cf0-9276-4ecc-a756-97dc76c0ac15"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:19.699547 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.699528 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c572cf0-9276-4ecc-a756-97dc76c0ac15-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:00:19.699626 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.699551 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6577c714-acaf-443a-8072-0dbb83e65811-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:00:19.699626 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.699566 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvsts\" (UniqueName: \"kubernetes.io/projected/5f825e65-da5a-4cfd-8bd6-8ab39093c889-kube-api-access-gvsts\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:00:19.701559 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.701521 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c572cf0-9276-4ecc-a756-97dc76c0ac15-kube-api-access-fwg8p" pod "3c572cf0-9276-4ecc-a756-97dc76c0ac15" (UID: "3c572cf0-9276-4ecc-a756-97dc76c0ac15"). InnerVolumeSpecName "kube-api-access-fwg8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:19.701679 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.701561 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6577c714-acaf-443a-8072-0dbb83e65811-kube-api-access-t6n6q" pod "6577c714-acaf-443a-8072-0dbb83e65811" (UID: "6577c714-acaf-443a-8072-0dbb83e65811"). InnerVolumeSpecName "kube-api-access-t6n6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:19.711178 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.711116 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6"] Oct 30 19:00:19.745491 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.745364 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-83d5bd-29889780-5pmh6"] Oct 30 19:00:19.801419 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.801376 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwg8p\" (UniqueName: \"kubernetes.io/projected/3c572cf0-9276-4ecc-a756-97dc76c0ac15-kube-api-access-fwg8p\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:00:19.801419 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:19.801418 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6n6q\" (UniqueName: \"kubernetes.io/projected/6577c714-acaf-443a-8072-0dbb83e65811-kube-api-access-t6n6q\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:00:20.101526 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.101489 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20f9e740-3071-4100-ad6e-30e89d622bca" path="/var/lib/kubelet/pods/20f9e740-3071-4100-ad6e-30e89d622bca/volumes" Oct 30 19:00:20.123388 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.123358 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz" Oct 30 19:00:20.123850 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.123356 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz" event={"ID":"3c572cf0-9276-4ecc-a756-97dc76c0ac15","Type":"ContainerDied","Data":"40e304a59a8a7eae68ae3fb5d80dde61a39fd7bb885f8a2d1742328a6d1aadb2"} Oct 30 19:00:20.123850 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.123512 4531 scope.go:122] "RemoveContainer" containerID="801b2b9c735b2da8b4bc8f72329c8e4759220747f9bb955e68485559948ec181" Oct 30 19:00:20.125442 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.125281 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc" event={"ID":"5f825e65-da5a-4cfd-8bd6-8ab39093c889","Type":"ContainerDied","Data":"b7e3d3d10d4673c2cfa77c2690a465afa0605d1b148d66cea202e004e9349534"} Oct 30 19:00:20.125442 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.125294 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc" Oct 30 19:00:20.128166 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.128118 4531 generic.go:356] "Generic (PLEG): container finished" podID="a03d7452-d8ee-472b-a1db-70ace6d017af" containerID="e7ecec941ae1cd840500e95c31962a94e6158f59a45960b7ab638415680a1816" exitCode=60 Oct 30 19:00:20.128285 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.128266 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72" event={"ID":"a03d7452-d8ee-472b-a1db-70ace6d017af","Type":"ContainerDied","Data":"e7ecec941ae1cd840500e95c31962a94e6158f59a45960b7ab638415680a1816"} Oct 30 19:00:20.128767 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.128743 4531 scope.go:122] "RemoveContainer" containerID="e7ecec941ae1cd840500e95c31962a94e6158f59a45960b7ab638415680a1816" Oct 30 19:00:20.129413 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:00:20.129255 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889780-jcj72_odh-ai-gateway-infra(a03d7452-d8ee-472b-a1db-70ace6d017af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72" podUID="a03d7452-d8ee-472b-a1db-70ace6d017af" Oct 30 19:00:20.133457 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.133427 4531 generic.go:356] "Generic (PLEG): container finished" podID="9424aef8-1e44-41a3-ae4a-30d48eed876c" containerID="d3d26eeb0796ca5ec8d0bfc03a2b4379383a366757d88be06ea6125a9162f173" exitCode=60 Oct 30 19:00:20.133580 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.133495 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p" event={"ID":"9424aef8-1e44-41a3-ae4a-30d48eed876c","Type":"ContainerDied","Data":"d3d26eeb0796ca5ec8d0bfc03a2b4379383a366757d88be06ea6125a9162f173"} Oct 30 19:00:20.133967 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.133948 4531 scope.go:122] "RemoveContainer" containerID="d3d26eeb0796ca5ec8d0bfc03a2b4379383a366757d88be06ea6125a9162f173" Oct 30 19:00:20.134373 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:00:20.134346 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p_odh-ai-gateway-infra(9424aef8-1e44-41a3-ae4a-30d48eed876c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p" podUID="9424aef8-1e44-41a3-ae4a-30d48eed876c" Oct 30 19:00:20.135395 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.135368 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd" event={"ID":"6577c714-acaf-443a-8072-0dbb83e65811","Type":"ContainerDied","Data":"9c83e3231c6d606e879888b91d243401551efb47c3982f2804dd9cf43733f5ef"} Oct 30 19:00:20.135395 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.135387 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd" Oct 30 19:00:20.137876 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.137850 4531 scope.go:122] "RemoveContainer" containerID="bb21c95ad14104211d484425c0355829eed6a83e93585c2c97bd0f83ea6d2d8a" Oct 30 19:00:20.150156 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.150098 4531 scope.go:122] "RemoveContainer" containerID="433f11593cc1b22a18256995422ec700d2afcef99fdd4002fdd67083dd60b431" Oct 30 19:00:20.167290 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.167265 4531 scope.go:122] "RemoveContainer" containerID="8cc58b7d4d8f775178110c9cfc5ec311ad7d6b2ad7ab948f1275d61e194aa06e" Oct 30 19:00:20.193656 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.193630 4531 scope.go:122] "RemoveContainer" containerID="95d1b63089e97985391c2ea061300e70c2a4261cf508fd984a7699298d36f049" Oct 30 19:00:20.296306 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.296275 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j" Oct 30 19:00:20.411250 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.411212 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0949f5d9-3405-4b6c-946e-8a707a605045-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0949f5d9-3405-4b6c-946e-8a707a605045-maas-api-service-ca\") pod \"0949f5d9-3405-4b6c-946e-8a707a605045\" (UID: \"0949f5d9-3405-4b6c-946e-8a707a605045\") " Oct 30 19:00:20.411492 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.411412 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0949f5d9-3405-4b6c-946e-8a707a605045-kube-api-access-rmpsn\" (UniqueName: \"kubernetes.io/projected/0949f5d9-3405-4b6c-946e-8a707a605045-kube-api-access-rmpsn\") pod \"0949f5d9-3405-4b6c-946e-8a707a605045\" (UID: \"0949f5d9-3405-4b6c-946e-8a707a605045\") " Oct 30 19:00:20.412065 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.412017 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0949f5d9-3405-4b6c-946e-8a707a605045-maas-api-service-ca" pod "0949f5d9-3405-4b6c-946e-8a707a605045" (UID: "0949f5d9-3405-4b6c-946e-8a707a605045"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:20.415715 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.415664 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0949f5d9-3405-4b6c-946e-8a707a605045-kube-api-access-rmpsn" pod "0949f5d9-3405-4b6c-946e-8a707a605045" (UID: "0949f5d9-3405-4b6c-946e-8a707a605045"). InnerVolumeSpecName "kube-api-access-rmpsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:20.444633 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.444590 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72"] Oct 30 19:00:20.513857 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.513815 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmpsn\" (UniqueName: \"kubernetes.io/projected/0949f5d9-3405-4b6c-946e-8a707a605045-kube-api-access-rmpsn\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:00:20.513857 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.513861 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0949f5d9-3405-4b6c-946e-8a707a605045-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:00:20.685636 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.685528 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz"] Oct 30 19:00:20.689485 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.689451 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889780-tjgcz"] Oct 30 19:00:20.966170 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.966033 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd"] Oct 30 19:00:20.972931 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:20.972894 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889780-7tqjd"] Oct 30 19:00:21.032239 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:21.032185 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p"] Oct 30 19:00:21.090867 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:21.090826 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc"] Oct 30 19:00:21.095523 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:21.095492 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2ade09-29889780-lmbzc"] Oct 30 19:00:21.160673 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:21.160646 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j" Oct 30 19:00:21.160673 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:21.160646 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j" event={"ID":"0949f5d9-3405-4b6c-946e-8a707a605045","Type":"ContainerDied","Data":"981550b6f59c63a675113c34729fb8343399e6482de0b0f66391e18df03195b4"} Oct 30 19:00:21.161166 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:21.160752 4531 scope.go:122] "RemoveContainer" containerID="8769ecc1e6742f94078f1fa731c6779a7293165350c031d2657433486dbb1054" Oct 30 19:00:21.200741 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:21.198878 4531 patch_prober.go:29] interesting pod/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:00:21.200741 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:21.198984 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" podUID="52581cb3-251f-4450-b981-1b7fa19d9367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:00:21.377924 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:21.377744 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72" Oct 30 19:00:21.415603 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:21.415573 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p" Oct 30 19:00:21.426697 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:21.426656 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9424aef8-1e44-41a3-ae4a-30d48eed876c-kube-api-access-2v97s\" (UniqueName: \"kubernetes.io/projected/9424aef8-1e44-41a3-ae4a-30d48eed876c-kube-api-access-2v97s\") pod \"9424aef8-1e44-41a3-ae4a-30d48eed876c\" (UID: \"9424aef8-1e44-41a3-ae4a-30d48eed876c\") " Oct 30 19:00:21.426916 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:21.426773 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a03d7452-d8ee-472b-a1db-70ace6d017af-kube-api-access-86rnh\" (UniqueName: \"kubernetes.io/projected/a03d7452-d8ee-472b-a1db-70ace6d017af-kube-api-access-86rnh\") pod \"a03d7452-d8ee-472b-a1db-70ace6d017af\" (UID: \"a03d7452-d8ee-472b-a1db-70ace6d017af\") " Oct 30 19:00:21.426916 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:21.426903 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a03d7452-d8ee-472b-a1db-70ace6d017af-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a03d7452-d8ee-472b-a1db-70ace6d017af-maas-api-service-ca\") pod \"a03d7452-d8ee-472b-a1db-70ace6d017af\" (UID: \"a03d7452-d8ee-472b-a1db-70ace6d017af\") " Oct 30 19:00:21.427044 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:21.426959 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9424aef8-1e44-41a3-ae4a-30d48eed876c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9424aef8-1e44-41a3-ae4a-30d48eed876c-maas-api-service-ca\") pod \"9424aef8-1e44-41a3-ae4a-30d48eed876c\" (UID: \"9424aef8-1e44-41a3-ae4a-30d48eed876c\") " Oct 30 19:00:21.427679 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:21.427643 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9424aef8-1e44-41a3-ae4a-30d48eed876c-maas-api-service-ca" pod "9424aef8-1e44-41a3-ae4a-30d48eed876c" (UID: "9424aef8-1e44-41a3-ae4a-30d48eed876c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:21.427819 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:21.427702 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a03d7452-d8ee-472b-a1db-70ace6d017af-maas-api-service-ca" pod "a03d7452-d8ee-472b-a1db-70ace6d017af" (UID: "a03d7452-d8ee-472b-a1db-70ace6d017af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:21.430069 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:21.430032 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9424aef8-1e44-41a3-ae4a-30d48eed876c-kube-api-access-2v97s" pod "9424aef8-1e44-41a3-ae4a-30d48eed876c" (UID: "9424aef8-1e44-41a3-ae4a-30d48eed876c"). InnerVolumeSpecName "kube-api-access-2v97s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:21.430241 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:21.430059 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03d7452-d8ee-472b-a1db-70ace6d017af-kube-api-access-86rnh" pod "a03d7452-d8ee-472b-a1db-70ace6d017af" (UID: "a03d7452-d8ee-472b-a1db-70ace6d017af"). InnerVolumeSpecName "kube-api-access-86rnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:21.529428 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:21.529382 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a03d7452-d8ee-472b-a1db-70ace6d017af-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:00:21.529428 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:21.529428 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9424aef8-1e44-41a3-ae4a-30d48eed876c-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:00:21.529657 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:21.529444 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v97s\" (UniqueName: \"kubernetes.io/projected/9424aef8-1e44-41a3-ae4a-30d48eed876c-kube-api-access-2v97s\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:00:21.529657 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:21.529461 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86rnh\" (UniqueName: \"kubernetes.io/projected/a03d7452-d8ee-472b-a1db-70ace6d017af-kube-api-access-86rnh\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:00:21.606167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:21.606102 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j"] Oct 30 19:00:21.618181 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:21.618123 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889780-2gt9j"] Oct 30 19:00:22.102045 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:22.102005 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0949f5d9-3405-4b6c-946e-8a707a605045" path="/var/lib/kubelet/pods/0949f5d9-3405-4b6c-946e-8a707a605045/volumes" Oct 30 19:00:22.102523 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:22.102506 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c572cf0-9276-4ecc-a756-97dc76c0ac15" path="/var/lib/kubelet/pods/3c572cf0-9276-4ecc-a756-97dc76c0ac15/volumes" Oct 30 19:00:22.102898 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:22.102885 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f825e65-da5a-4cfd-8bd6-8ab39093c889" path="/var/lib/kubelet/pods/5f825e65-da5a-4cfd-8bd6-8ab39093c889/volumes" Oct 30 19:00:22.103644 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:22.103627 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6577c714-acaf-443a-8072-0dbb83e65811" path="/var/lib/kubelet/pods/6577c714-acaf-443a-8072-0dbb83e65811/volumes" Oct 30 19:00:22.178421 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:22.177412 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72" event={"ID":"a03d7452-d8ee-472b-a1db-70ace6d017af","Type":"ContainerDied","Data":"4ca0a2dca44d8f68b2aa042d7c3d554eda99068b1ba237143902746e9a545738"} Oct 30 19:00:22.178421 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:22.177493 4531 scope.go:122] "RemoveContainer" containerID="e7ecec941ae1cd840500e95c31962a94e6158f59a45960b7ab638415680a1816" Oct 30 19:00:22.178421 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:22.177602 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72" Oct 30 19:00:22.184802 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:22.184182 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p" Oct 30 19:00:22.184802 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:22.184200 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p" event={"ID":"9424aef8-1e44-41a3-ae4a-30d48eed876c","Type":"ContainerDied","Data":"c2eaf8eb284057d2c6c6d866d8ca5621bcc35b108d9c13cf09593bc25079de41"} Oct 30 19:00:22.193251 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:22.193115 4531 scope.go:122] "RemoveContainer" containerID="d3d26eeb0796ca5ec8d0bfc03a2b4379383a366757d88be06ea6125a9162f173" Oct 30 19:00:22.370739 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:22.370626 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72"] Oct 30 19:00:22.374845 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:22.374805 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889780-jcj72"] Oct 30 19:00:22.497292 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:22.497253 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p"] Oct 30 19:00:22.498949 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:22.498922 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-132af1-29889780-6zg4p"] Oct 30 19:00:24.101605 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:24.101568 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9424aef8-1e44-41a3-ae4a-30d48eed876c" path="/var/lib/kubelet/pods/9424aef8-1e44-41a3-ae4a-30d48eed876c/volumes" Oct 30 19:00:24.102104 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:24.102058 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a03d7452-d8ee-472b-a1db-70ace6d017af" path="/var/lib/kubelet/pods/a03d7452-d8ee-472b-a1db-70ace6d017af/volumes" Oct 30 19:00:24.203487 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:24.203449 4531 generic.go:356] "Generic (PLEG): container finished" podID="52581cb3-251f-4450-b981-1b7fa19d9367" containerID="28de017c8644ce0e679c153d9a2e123e3d8940c1bfb92cec5b413eae59930edc" exitCode=0 Oct 30 19:00:24.203679 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:24.203531 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" event={"ID":"52581cb3-251f-4450-b981-1b7fa19d9367","Type":"ContainerDied","Data":"28de017c8644ce0e679c153d9a2e123e3d8940c1bfb92cec5b413eae59930edc"} Oct 30 19:00:24.204017 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:24.203999 4531 scope.go:122] "RemoveContainer" containerID="28de017c8644ce0e679c153d9a2e123e3d8940c1bfb92cec5b413eae59930edc" Oct 30 19:00:24.205655 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:24.205624 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:00:24.205726 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:24.205698 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:00:24.205763 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:24.205729 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:00:25.214513 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:25.214478 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" event={"ID":"52581cb3-251f-4450-b981-1b7fa19d9367","Type":"ContainerStarted","Data":"83f1bbe34104c814717834a5b147f817d54fa785254909f4e8652160c93f437f"} Oct 30 19:00:25.374868 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:25.374832 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 19:00:25.375184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:25.375120 4531 patch_prober.go:29] interesting pod/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 19:00:25.375296 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:25.375228 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" podUID="52581cb3-251f-4450-b981-1b7fa19d9367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 19:00:26.374375 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:26.374340 4531 patch_prober.go:29] interesting pod/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 19:00:26.374819 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:26.374406 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" podUID="52581cb3-251f-4450-b981-1b7fa19d9367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 19:00:26.402213 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:00:26.402174 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:00:27.374354 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:27.374315 4531 patch_prober.go:29] interesting pod/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 19:00:27.374588 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:27.374376 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" podUID="52581cb3-251f-4450-b981-1b7fa19d9367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 19:00:28.374899 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:28.374857 4531 patch_prober.go:29] interesting pod/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 19:00:28.375474 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:28.374923 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" podUID="52581cb3-251f-4450-b981-1b7fa19d9367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 19:00:29.374982 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:29.374940 4531 patch_prober.go:29] interesting pod/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 19:00:29.375475 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:29.375006 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" podUID="52581cb3-251f-4450-b981-1b7fa19d9367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 19:00:30.374574 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:30.374534 4531 patch_prober.go:29] interesting pod/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 19:00:30.374778 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:30.374608 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" podUID="52581cb3-251f-4450-b981-1b7fa19d9367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 19:00:31.379850 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:31.379816 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 19:00:31.380287 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:31.380270 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 19:00:31.381035 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:31.381018 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 19:00:35.899151 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899087 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f1ed4d8-ft2d5"] Oct 30 19:00:35.899724 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899707 4531 cpu_manager.go:414] "containerMap: removing container" podUID="20f9e740-3071-4100-ad6e-30e89d622bca" containerName="cleanup" Oct 30 19:00:35.899724 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899724 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20f9e740-3071-4100-ad6e-30e89d622bca" containerName="cleanup" Oct 30 19:00:35.899845 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899733 4531 cpu_manager.go:414] "containerMap: removing container" podUID="5f825e65-da5a-4cfd-8bd6-8ab39093c889" containerName="cleanup" Oct 30 19:00:35.899845 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899739 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f825e65-da5a-4cfd-8bd6-8ab39093c889" containerName="cleanup" Oct 30 19:00:35.899845 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899749 4531 cpu_manager.go:414] "containerMap: removing container" podUID="0949f5d9-3405-4b6c-946e-8a707a605045" containerName="cleanup" Oct 30 19:00:35.899845 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899754 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0949f5d9-3405-4b6c-946e-8a707a605045" containerName="cleanup" Oct 30 19:00:35.899845 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899770 4531 cpu_manager.go:414] "containerMap: removing container" podUID="20f9e740-3071-4100-ad6e-30e89d622bca" containerName="cleanup" Oct 30 19:00:35.899845 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899775 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20f9e740-3071-4100-ad6e-30e89d622bca" containerName="cleanup" Oct 30 19:00:35.899845 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899783 4531 cpu_manager.go:414] "containerMap: removing container" podUID="6577c714-acaf-443a-8072-0dbb83e65811" containerName="cleanup" Oct 30 19:00:35.899845 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899787 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6577c714-acaf-443a-8072-0dbb83e65811" containerName="cleanup" Oct 30 19:00:35.899845 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899795 4531 cpu_manager.go:414] "containerMap: removing container" podUID="20f9e740-3071-4100-ad6e-30e89d622bca" containerName="cleanup" Oct 30 19:00:35.899845 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899800 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20f9e740-3071-4100-ad6e-30e89d622bca" containerName="cleanup" Oct 30 19:00:35.899845 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899807 4531 cpu_manager.go:414] "containerMap: removing container" podUID="9424aef8-1e44-41a3-ae4a-30d48eed876c" containerName="cleanup" Oct 30 19:00:35.899845 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899811 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9424aef8-1e44-41a3-ae4a-30d48eed876c" containerName="cleanup" Oct 30 19:00:35.899845 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899823 4531 cpu_manager.go:414] "containerMap: removing container" podUID="3c572cf0-9276-4ecc-a756-97dc76c0ac15" containerName="cleanup" Oct 30 19:00:35.899845 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899827 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c572cf0-9276-4ecc-a756-97dc76c0ac15" containerName="cleanup" Oct 30 19:00:35.899845 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899835 4531 cpu_manager.go:414] "containerMap: removing container" podUID="6577c714-acaf-443a-8072-0dbb83e65811" containerName="cleanup" Oct 30 19:00:35.899845 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899839 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6577c714-acaf-443a-8072-0dbb83e65811" containerName="cleanup" Oct 30 19:00:35.899845 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899847 4531 cpu_manager.go:414] "containerMap: removing container" podUID="5f825e65-da5a-4cfd-8bd6-8ab39093c889" containerName="cleanup" Oct 30 19:00:35.899845 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899852 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f825e65-da5a-4cfd-8bd6-8ab39093c889" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899858 4531 cpu_manager.go:414] "containerMap: removing container" podUID="a03d7452-d8ee-472b-a1db-70ace6d017af" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899862 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a03d7452-d8ee-472b-a1db-70ace6d017af" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899870 4531 cpu_manager.go:414] "containerMap: removing container" podUID="3c572cf0-9276-4ecc-a756-97dc76c0ac15" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899874 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c572cf0-9276-4ecc-a756-97dc76c0ac15" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899882 4531 cpu_manager.go:414] "containerMap: removing container" podUID="a03d7452-d8ee-472b-a1db-70ace6d017af" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899886 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a03d7452-d8ee-472b-a1db-70ace6d017af" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899900 4531 cpu_manager.go:414] "containerMap: removing container" podUID="9424aef8-1e44-41a3-ae4a-30d48eed876c" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899906 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9424aef8-1e44-41a3-ae4a-30d48eed876c" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899912 4531 cpu_manager.go:414] "containerMap: removing container" podUID="9424aef8-1e44-41a3-ae4a-30d48eed876c" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899916 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9424aef8-1e44-41a3-ae4a-30d48eed876c" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899925 4531 cpu_manager.go:414] "containerMap: removing container" podUID="3c572cf0-9276-4ecc-a756-97dc76c0ac15" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899929 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c572cf0-9276-4ecc-a756-97dc76c0ac15" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899934 4531 cpu_manager.go:414] "containerMap: removing container" podUID="0949f5d9-3405-4b6c-946e-8a707a605045" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899938 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0949f5d9-3405-4b6c-946e-8a707a605045" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899945 4531 cpu_manager.go:414] "containerMap: removing container" podUID="a03d7452-d8ee-472b-a1db-70ace6d017af" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.899949 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a03d7452-d8ee-472b-a1db-70ace6d017af" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.900028 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6577c714-acaf-443a-8072-0dbb83e65811" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.900036 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f825e65-da5a-4cfd-8bd6-8ab39093c889" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.900041 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a03d7452-d8ee-472b-a1db-70ace6d017af" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.900047 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c572cf0-9276-4ecc-a756-97dc76c0ac15" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.900051 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a03d7452-d8ee-472b-a1db-70ace6d017af" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.900058 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0949f5d9-3405-4b6c-946e-8a707a605045" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.900065 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9424aef8-1e44-41a3-ae4a-30d48eed876c" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.900072 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6577c714-acaf-443a-8072-0dbb83e65811" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.900076 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9424aef8-1e44-41a3-ae4a-30d48eed876c" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.900081 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0949f5d9-3405-4b6c-946e-8a707a605045" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.900087 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a03d7452-d8ee-472b-a1db-70ace6d017af" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.900094 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="20f9e740-3071-4100-ad6e-30e89d622bca" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.900099 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f825e65-da5a-4cfd-8bd6-8ab39093c889" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.900104 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0949f5d9-3405-4b6c-946e-8a707a605045" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.900111 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="20f9e740-3071-4100-ad6e-30e89d622bca" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.900117 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="20f9e740-3071-4100-ad6e-30e89d622bca" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.900123 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9424aef8-1e44-41a3-ae4a-30d48eed876c" containerName="cleanup" Oct 30 19:00:35.900712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.900145 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c572cf0-9276-4ecc-a756-97dc76c0ac15" containerName="cleanup" Oct 30 19:00:35.904683 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.904662 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1ed4d8-ft2d5" Oct 30 19:00:35.965916 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:35.965884 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:00:36.015247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:36.015209 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:00:36.015463 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:36.015265 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:00:36.015463 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:36.015304 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:00:36.019545 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:36.019514 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f1ed4d8-ft2d5"] Oct 30 19:00:36.062555 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:36.062514 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f1ed4d8-ft2d5\" (UID: \"6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1ed4d8-ft2d5" Oct 30 19:00:36.062775 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:36.062750 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f1ed4d8-ft2d5\" (UID: \"6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1ed4d8-ft2d5" Oct 30 19:00:36.062829 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:36.062798 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4jsb\" (UniqueName: \"kubernetes.io/projected/6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6-kube-api-access-n4jsb\") pod \"maas-api-revoke-sub-3f1ed4d8-ft2d5\" (UID: \"6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1ed4d8-ft2d5" Oct 30 19:00:36.164315 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:36.164214 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f1ed4d8-ft2d5\" (UID: \"6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1ed4d8-ft2d5" Oct 30 19:00:36.164473 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:36.164337 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4jsb\" (UniqueName: \"kubernetes.io/projected/6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6-kube-api-access-n4jsb\") pod \"maas-api-revoke-sub-3f1ed4d8-ft2d5\" (UID: \"6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1ed4d8-ft2d5" Oct 30 19:00:36.164549 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:36.164478 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f1ed4d8-ft2d5\" (UID: \"6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1ed4d8-ft2d5" Oct 30 19:00:36.165231 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:36.165199 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f1ed4d8-ft2d5\" (UID: \"6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1ed4d8-ft2d5" Oct 30 19:00:36.167208 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:36.167188 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f1ed4d8-ft2d5\" (UID: \"6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1ed4d8-ft2d5" Oct 30 19:00:36.225917 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:36.225886 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4jsb\" (UniqueName: \"kubernetes.io/projected/6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6-kube-api-access-n4jsb\") pod \"maas-api-revoke-sub-3f1ed4d8-ft2d5\" (UID: \"6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1ed4d8-ft2d5" Oct 30 19:00:36.518688 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:36.518646 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1ed4d8-ft2d5" Oct 30 19:00:36.900245 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:36.900204 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f1ed4d8-ft2d5"] Oct 30 19:00:36.901290 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:00:36.901257 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b90edb7_702f_45e4_b7d7_fdbe38ebfaf6.slice/crio-a7a8b7927d9ad7dc02f69e2a551ed5f54a731fc48d372396535fdac3c9faecdf WatchSource:0}: Error finding container a7a8b7927d9ad7dc02f69e2a551ed5f54a731fc48d372396535fdac3c9faecdf: Status 404 returned error can't find the container with id a7a8b7927d9ad7dc02f69e2a551ed5f54a731fc48d372396535fdac3c9faecdf Oct 30 19:00:37.336304 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:37.336266 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1ed4d8-ft2d5" event={"ID":"6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6","Type":"ContainerStarted","Data":"2ac379e71ba63f0d8be09f514d364c0597ab745d7c3f93b205a942d458d39e67"} Oct 30 19:00:37.336495 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:37.336324 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1ed4d8-ft2d5" event={"ID":"6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6","Type":"ContainerStarted","Data":"a7a8b7927d9ad7dc02f69e2a551ed5f54a731fc48d372396535fdac3c9faecdf"} Oct 30 19:00:37.461914 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:37.461856 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1ed4d8-ft2d5" podStartSLOduration=2.461838157 podStartE2EDuration="2.461838157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:37.459698337 +0000 UTC m=+1597.486803298" watchObservedRunningTime="2026-10-30 19:00:37.461838157 +0000 UTC m=+1597.488943115" Oct 30 19:00:38.345261 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:38.345226 4531 generic.go:356] "Generic (PLEG): container finished" podID="6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6" containerID="2ac379e71ba63f0d8be09f514d364c0597ab745d7c3f93b205a942d458d39e67" exitCode=0 Oct 30 19:00:38.345770 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:38.345318 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1ed4d8-ft2d5" event={"ID":"6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6","Type":"ContainerDied","Data":"2ac379e71ba63f0d8be09f514d364c0597ab745d7c3f93b205a942d458d39e67"} Oct 30 19:00:39.521597 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:39.521568 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1ed4d8-ft2d5" Oct 30 19:00:39.613185 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:39.613026 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6-maas-api-xks-ca\") pod \"6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6\" (UID: \"6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6\") " Oct 30 19:00:39.613392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:39.613281 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6-kube-api-access-n4jsb\" (UniqueName: \"kubernetes.io/projected/6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6-kube-api-access-n4jsb\") pod \"6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6\" (UID: \"6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6\") " Oct 30 19:00:39.613392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:39.613343 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6-maas-api-openshift-ca\") pod \"6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6\" (UID: \"6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6\") " Oct 30 19:00:39.613814 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:39.613784 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6-maas-api-openshift-ca" pod "6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6" (UID: "6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:39.615579 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:39.615542 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6-maas-api-xks-ca" pod "6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6" (UID: "6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:39.615579 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:39.615571 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6-kube-api-access-n4jsb" pod "6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6" (UID: "6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6"). InnerVolumeSpecName "kube-api-access-n4jsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:39.714821 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:39.714782 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4jsb\" (UniqueName: \"kubernetes.io/projected/6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6-kube-api-access-n4jsb\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:00:39.714821 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:39.714818 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:00:39.714821 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:39.714831 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:00:40.361762 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:40.361674 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1ed4d8-ft2d5" Oct 30 19:00:40.361762 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:40.361715 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1ed4d8-ft2d5" event={"ID":"6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6","Type":"ContainerDied","Data":"a7a8b7927d9ad7dc02f69e2a551ed5f54a731fc48d372396535fdac3c9faecdf"} Oct 30 19:00:40.361960 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:40.361775 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7a8b7927d9ad7dc02f69e2a551ed5f54a731fc48d372396535fdac3c9faecdf" Oct 30 19:00:45.052546 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:45.052509 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ef8cbc-vmgh7"] Oct 30 19:00:45.101516 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:45.101483 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ef8cbc-vmgh7"] Oct 30 19:00:46.100440 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:46.100403 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97fa301a-7f41-4d7c-8ba4-f67ad32f256a" path="/var/lib/kubelet/pods/97fa301a-7f41-4d7c-8ba4-f67ad32f256a/volumes" Oct 30 19:00:55.680267 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.680229 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q"] Oct 30 19:00:55.680888 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.680869 4531 cpu_manager.go:414] "containerMap: removing container" podUID="6577c714-acaf-443a-8072-0dbb83e65811" containerName="cleanup" Oct 30 19:00:55.680932 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.680892 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6577c714-acaf-443a-8072-0dbb83e65811" containerName="cleanup" Oct 30 19:00:55.680932 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.680911 4531 cpu_manager.go:414] "containerMap: removing container" podUID="5f825e65-da5a-4cfd-8bd6-8ab39093c889" containerName="cleanup" Oct 30 19:00:55.680932 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.680918 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f825e65-da5a-4cfd-8bd6-8ab39093c889" containerName="cleanup" Oct 30 19:00:55.681030 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.680961 4531 cpu_manager.go:414] "containerMap: removing container" podUID="6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6" containerName="revoke-keys" Oct 30 19:00:55.681030 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.680969 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6" containerName="revoke-keys" Oct 30 19:00:55.681030 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.680987 4531 cpu_manager.go:414] "containerMap: removing container" podUID="0949f5d9-3405-4b6c-946e-8a707a605045" containerName="cleanup" Oct 30 19:00:55.681030 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.680995 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0949f5d9-3405-4b6c-946e-8a707a605045" containerName="cleanup" Oct 30 19:00:55.681235 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.681114 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6" containerName="revoke-keys" Oct 30 19:00:55.681235 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.681150 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f825e65-da5a-4cfd-8bd6-8ab39093c889" containerName="cleanup" Oct 30 19:00:55.681235 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.681159 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c572cf0-9276-4ecc-a756-97dc76c0ac15" containerName="cleanup" Oct 30 19:00:55.681235 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.681170 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6577c714-acaf-443a-8072-0dbb83e65811" containerName="cleanup" Oct 30 19:00:55.684879 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.684850 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:55.798408 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.798361 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-token\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:55.798603 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.798521 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-workload-certs\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:55.798683 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.798659 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-envoy\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:55.798764 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.798746 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqkzg\" (UniqueName: \"kubernetes.io/projected/0e048745-6a7f-4d2f-a9f8-768e7d411370-kube-api-access-pqkzg\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:55.798817 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.798782 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-credential-socket\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:55.798900 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.798884 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-podinfo\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:55.798974 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.798946 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e048745-6a7f-4d2f-a9f8-768e7d411370-istiod-ca-cert\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:55.799046 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.799029 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-workload-socket\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:55.799105 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.799076 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-data\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:55.867214 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.867182 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-66baf45c-openshift-default-dockercfg-sqvln\"" Oct 30 19:00:55.900526 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.900480 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-token\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:55.900728 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.900565 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-workload-certs\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:55.900728 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.900622 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-envoy\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:55.900728 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.900671 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqkzg\" (UniqueName: \"kubernetes.io/projected/0e048745-6a7f-4d2f-a9f8-768e7d411370-kube-api-access-pqkzg\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:55.900728 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.900696 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-credential-socket\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:55.900898 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.900731 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-podinfo\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:55.900898 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.900754 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e048745-6a7f-4d2f-a9f8-768e7d411370-istiod-ca-cert\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:55.900898 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.900780 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-workload-socket\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:55.900898 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.900802 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-data\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:55.901098 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.901075 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-workload-certs\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:55.901260 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.901237 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-data\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:55.901356 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.901306 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-credential-socket\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:55.901356 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.901344 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-workload-socket\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:55.901775 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.901750 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e048745-6a7f-4d2f-a9f8-768e7d411370-istiod-ca-cert\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:55.903507 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.903485 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-podinfo\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:55.903608 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.903534 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-envoy\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:55.918537 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.918494 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q"] Oct 30 19:00:55.985229 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:55.985150 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqkzg\" (UniqueName: \"kubernetes.io/projected/0e048745-6a7f-4d2f-a9f8-768e7d411370-kube-api-access-pqkzg\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:56.014773 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:56.014734 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-token\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:56.301168 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:56.301092 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:56.420905 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:00:56.420860 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:00:56.741717 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:56.741375 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q"] Oct 30 19:00:56.742114 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:00:56.741888 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e048745_6a7f_4d2f_a9f8_768e7d411370.slice/crio-9a1beb1ed9abe38c6897e6019c14cc225a86e257fda71221730f6947b6fed578 WatchSource:0}: Error finding container 9a1beb1ed9abe38c6897e6019c14cc225a86e257fda71221730f6947b6fed578: Status 404 returned error can't find the container with id 9a1beb1ed9abe38c6897e6019c14cc225a86e257fda71221730f6947b6fed578 Oct 30 19:00:56.744881 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:56.744842 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:00:56.745012 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:56.744913 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:00:56.745012 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:56.744947 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:00:57.106891 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:57.106843 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-85ab6a-mql96"] Oct 30 19:00:57.111304 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:57.111274 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-85ab6a-mql96" Oct 30 19:00:57.167968 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:57.167902 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:00:57.167968 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:57.167902 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:00:57.172771 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:57.172731 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-85ab6a-mql96"] Oct 30 19:00:57.215725 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:57.215688 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:00:57.215936 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:57.215770 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:00:57.218619 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:57.218561 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb908afb-d997-4479-997f-be8958658b69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-85ab6a-mql96\" (UID: \"eb908afb-d997-4479-997f-be8958658b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-85ab6a-mql96" Oct 30 19:00:57.218850 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:57.218823 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb908afb-d997-4479-997f-be8958658b69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-85ab6a-mql96\" (UID: \"eb908afb-d997-4479-997f-be8958658b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-85ab6a-mql96" Oct 30 19:00:57.219018 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:57.218976 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjm2j\" (UniqueName: \"kubernetes.io/projected/eb908afb-d997-4479-997f-be8958658b69-kube-api-access-pjm2j\") pod \"maas-api-revoke-keys-e2e-api-a-85ab6a-mql96\" (UID: \"eb908afb-d997-4479-997f-be8958658b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-85ab6a-mql96" Oct 30 19:00:57.321092 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:57.321029 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjm2j\" (UniqueName: \"kubernetes.io/projected/eb908afb-d997-4479-997f-be8958658b69-kube-api-access-pjm2j\") pod \"maas-api-revoke-keys-e2e-api-a-85ab6a-mql96\" (UID: \"eb908afb-d997-4479-997f-be8958658b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-85ab6a-mql96" Oct 30 19:00:57.321437 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:57.321407 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb908afb-d997-4479-997f-be8958658b69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-85ab6a-mql96\" (UID: \"eb908afb-d997-4479-997f-be8958658b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-85ab6a-mql96" Oct 30 19:00:57.321629 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:57.321610 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb908afb-d997-4479-997f-be8958658b69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-85ab6a-mql96\" (UID: \"eb908afb-d997-4479-997f-be8958658b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-85ab6a-mql96" Oct 30 19:00:57.322332 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:57.322301 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb908afb-d997-4479-997f-be8958658b69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-85ab6a-mql96\" (UID: \"eb908afb-d997-4479-997f-be8958658b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-85ab6a-mql96" Oct 30 19:00:57.324717 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:57.324678 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb908afb-d997-4479-997f-be8958658b69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-85ab6a-mql96\" (UID: \"eb908afb-d997-4479-997f-be8958658b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-85ab6a-mql96" Oct 30 19:00:57.382828 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:57.382740 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjm2j\" (UniqueName: \"kubernetes.io/projected/eb908afb-d997-4479-997f-be8958658b69-kube-api-access-pjm2j\") pod \"maas-api-revoke-keys-e2e-api-a-85ab6a-mql96\" (UID: \"eb908afb-d997-4479-997f-be8958658b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-85ab6a-mql96" Oct 30 19:00:57.428154 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:57.427311 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-85ab6a-mql96" Oct 30 19:00:57.559371 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:57.559334 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" event={"ID":"0e048745-6a7f-4d2f-a9f8-768e7d411370","Type":"ContainerStarted","Data":"6b96bfb1a78aa999315d4a41893d9acfb38ffdde8f69528a84724a4d8ec50bfb"} Oct 30 19:00:57.559576 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:57.559399 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" event={"ID":"0e048745-6a7f-4d2f-a9f8-768e7d411370","Type":"ContainerStarted","Data":"9a1beb1ed9abe38c6897e6019c14cc225a86e257fda71221730f6947b6fed578"} Oct 30 19:00:57.682580 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:57.682428 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" podStartSLOduration=2.682407189 podStartE2EDuration="2.682407189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:57.681580104 +0000 UTC m=+1617.708685074" watchObservedRunningTime="2026-10-30 19:00:57.682407189 +0000 UTC m=+1617.709512149" Oct 30 19:00:57.844475 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:57.844428 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-85ab6a-mql96"] Oct 30 19:00:57.848846 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:00:57.848812 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb908afb_d997_4479_997f_be8958658b69.slice/crio-905cc812d9c1c5bfce28862a04ae82c04362bd83f0b9a2461cd96a44a61ff7e8 WatchSource:0}: Error finding container 905cc812d9c1c5bfce28862a04ae82c04362bd83f0b9a2461cd96a44a61ff7e8: Status 404 returned error can't find the container with id 905cc812d9c1c5bfce28862a04ae82c04362bd83f0b9a2461cd96a44a61ff7e8 Oct 30 19:00:58.301914 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:58.301820 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:00:58.303522 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:58.303489 4531 patch_prober.go:29] interesting pod/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 19:00:58.303666 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:58.303547 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" podUID="0e048745-6a7f-4d2f-a9f8-768e7d411370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 19:00:58.569033 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:58.568941 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-85ab6a-mql96" event={"ID":"eb908afb-d997-4479-997f-be8958658b69","Type":"ContainerStarted","Data":"74e06388324273e2ce5886265df6ff3c31108a25470ca3265053d93c43a1e813"} Oct 30 19:00:58.569033 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:58.568997 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-85ab6a-mql96" event={"ID":"eb908afb-d997-4479-997f-be8958658b69","Type":"ContainerStarted","Data":"905cc812d9c1c5bfce28862a04ae82c04362bd83f0b9a2461cd96a44a61ff7e8"} Oct 30 19:00:58.692849 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:58.692795 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-85ab6a-mql96" podStartSLOduration=1.69277674 podStartE2EDuration="1.69277674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:58.689966334 +0000 UTC m=+1618.717071303" watchObservedRunningTime="2026-10-30 19:00:58.69277674 +0000 UTC m=+1618.719881698" Oct 30 19:00:59.301749 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:59.301711 4531 patch_prober.go:29] interesting pod/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 19:00:59.302207 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:59.301788 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" podUID="0e048745-6a7f-4d2f-a9f8-768e7d411370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 19:00:59.578075 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:59.577975 4531 generic.go:356] "Generic (PLEG): container finished" podID="eb908afb-d997-4479-997f-be8958658b69" containerID="74e06388324273e2ce5886265df6ff3c31108a25470ca3265053d93c43a1e813" exitCode=0 Oct 30 19:00:59.578298 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:00:59.578065 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-85ab6a-mql96" event={"ID":"eb908afb-d997-4479-997f-be8958658b69","Type":"ContainerDied","Data":"74e06388324273e2ce5886265df6ff3c31108a25470ca3265053d93c43a1e813"} Oct 30 19:01:00.302391 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:00.302350 4531 patch_prober.go:29] interesting pod/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 19:01:00.302812 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:00.302412 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" podUID="0e048745-6a7f-4d2f-a9f8-768e7d411370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 19:01:00.853589 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:00.853505 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-85ab6a-mql96" Oct 30 19:01:00.885979 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:00.885933 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb908afb-d997-4479-997f-be8958658b69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb908afb-d997-4479-997f-be8958658b69-maas-api-openshift-ca\") pod \"eb908afb-d997-4479-997f-be8958658b69\" (UID: \"eb908afb-d997-4479-997f-be8958658b69\") " Oct 30 19:01:00.886198 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:00.886024 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb908afb-d997-4479-997f-be8958658b69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb908afb-d997-4479-997f-be8958658b69-maas-api-xks-ca\") pod \"eb908afb-d997-4479-997f-be8958658b69\" (UID: \"eb908afb-d997-4479-997f-be8958658b69\") " Oct 30 19:01:00.886198 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:00.886190 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb908afb-d997-4479-997f-be8958658b69-kube-api-access-pjm2j\" (UniqueName: \"kubernetes.io/projected/eb908afb-d997-4479-997f-be8958658b69-kube-api-access-pjm2j\") pod \"eb908afb-d997-4479-997f-be8958658b69\" (UID: \"eb908afb-d997-4479-997f-be8958658b69\") " Oct 30 19:01:00.886462 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:00.886430 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb908afb-d997-4479-997f-be8958658b69-maas-api-openshift-ca" pod "eb908afb-d997-4479-997f-be8958658b69" (UID: "eb908afb-d997-4479-997f-be8958658b69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:01:00.886815 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:00.886724 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb908afb-d997-4479-997f-be8958658b69-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:01:00.888546 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:00.888512 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb908afb-d997-4479-997f-be8958658b69-maas-api-xks-ca" pod "eb908afb-d997-4479-997f-be8958658b69" (UID: "eb908afb-d997-4479-997f-be8958658b69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:01:00.888666 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:00.888580 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb908afb-d997-4479-997f-be8958658b69-kube-api-access-pjm2j" pod "eb908afb-d997-4479-997f-be8958658b69" (UID: "eb908afb-d997-4479-997f-be8958658b69"). InnerVolumeSpecName "kube-api-access-pjm2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:00.944091 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:00.944039 4531 scope.go:122] "RemoveContainer" containerID="cc613af989d7c13e5486361c0c41816081a131c176916c5a4ee3b2985ad8e8ba" Oct 30 19:01:00.956351 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:00.956325 4531 scope.go:122] "RemoveContainer" containerID="79fc3aec0fb617f9afeddf0a4836fed0764ecb508fe4c004fe547fd27b093a8b" Oct 30 19:01:00.967029 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:00.966992 4531 scope.go:122] "RemoveContainer" containerID="7e3c6512fc690ee217a6df05cd20929d235d3ba712f5ffc1777b0ab700cfb06e" Oct 30 19:01:00.980732 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:00.980698 4531 scope.go:122] "RemoveContainer" containerID="1ed82a1b3e43af1fb84f31c07aea0d8ba3fd6db635326e0c554e000c1c4eb7e8" Oct 30 19:01:00.988048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:00.988007 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjm2j\" (UniqueName: \"kubernetes.io/projected/eb908afb-d997-4479-997f-be8958658b69-kube-api-access-pjm2j\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:01:00.988048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:00.988050 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb908afb-d997-4479-997f-be8958658b69-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:01:00.996899 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:00.996859 4531 scope.go:122] "RemoveContainer" containerID="aaa59c807cea9f42da9de592e9c971f0a81a1d00297527569a066be2923770a6" Oct 30 19:01:01.008705 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:01.008684 4531 scope.go:122] "RemoveContainer" containerID="6781778bbf00a6d79a11cfd00da67a11c6444e3edc0c0aaf86c2554e09b12275" Oct 30 19:01:01.060147 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:01.060087 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac7496fa-jbqbv"] Oct 30 19:01:01.066166 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:01.066112 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac7496fa-jbqbv"] Oct 30 19:01:01.301949 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:01.301907 4531 patch_prober.go:29] interesting pod/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 19:01:01.302177 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:01.301968 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" podUID="0e048745-6a7f-4d2f-a9f8-768e7d411370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 19:01:01.593696 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:01.593664 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-85ab6a-mql96" Oct 30 19:01:01.594165 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:01.593692 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-85ab6a-mql96" event={"ID":"eb908afb-d997-4479-997f-be8958658b69","Type":"ContainerDied","Data":"905cc812d9c1c5bfce28862a04ae82c04362bd83f0b9a2461cd96a44a61ff7e8"} Oct 30 19:01:01.594165 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:01.593750 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="905cc812d9c1c5bfce28862a04ae82c04362bd83f0b9a2461cd96a44a61ff7e8" Oct 30 19:01:02.100013 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:02.099974 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="151d129d-222e-4c7b-91f1-828d24137732" path="/var/lib/kubelet/pods/151d129d-222e-4c7b-91f1-828d24137732/volumes" Oct 30 19:01:02.302325 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:02.302287 4531 patch_prober.go:29] interesting pod/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 19:01:02.302479 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:02.302343 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" podUID="0e048745-6a7f-4d2f-a9f8-768e7d411370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 19:01:03.302607 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:03.302565 4531 patch_prober.go:29] interesting pod/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 19:01:03.303151 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:03.302639 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" podUID="0e048745-6a7f-4d2f-a9f8-768e7d411370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 19:01:04.307777 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:04.307745 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:01:04.308269 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:04.308254 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:01:04.309014 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:04.308992 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:01:08.644669 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:08.644636 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj"] Oct 30 19:01:08.755200 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:08.755116 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7"] Oct 30 19:01:10.802760 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:10.802715 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p"] Oct 30 19:01:10.803406 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:10.803384 4531 cpu_manager.go:414] "containerMap: removing container" podUID="eb908afb-d997-4479-997f-be8958658b69" containerName="revoke-keys" Oct 30 19:01:10.803406 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:10.803406 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb908afb-d997-4479-997f-be8958658b69" containerName="revoke-keys" Oct 30 19:01:10.803522 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:10.803511 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb908afb-d997-4479-997f-be8958658b69" containerName="revoke-keys" Oct 30 19:01:10.809346 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:10.809318 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:10.968040 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:10.967995 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-workload-certs\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:10.968358 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:10.968327 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-token\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:10.968486 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:10.968400 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-podinfo\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:10.968486 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:10.968423 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-data\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:10.968486 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:10.968467 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-credential-socket\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:10.968711 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:10.968497 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-envoy\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:10.968711 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:10.968653 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7xjg\" (UniqueName: \"kubernetes.io/projected/8085a328-5dcc-44f9-9b91-6933bbbdd301-kube-api-access-x7xjg\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:10.968831 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:10.968727 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-workload-socket\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:10.968831 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:10.968794 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8085a328-5dcc-44f9-9b91-6933bbbdd301-istiod-ca-cert\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:11.030602 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:11.030562 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p"] Oct 30 19:01:11.069809 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:11.069711 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-credential-socket\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:11.069809 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:11.069753 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-envoy\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:11.070036 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:11.069845 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7xjg\" (UniqueName: \"kubernetes.io/projected/8085a328-5dcc-44f9-9b91-6933bbbdd301-kube-api-access-x7xjg\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:11.070036 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:11.069973 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-workload-socket\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:11.070036 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:11.070028 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8085a328-5dcc-44f9-9b91-6933bbbdd301-istiod-ca-cert\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:11.070258 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:11.070100 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-workload-certs\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:11.070258 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:11.070194 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-token\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:11.070370 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:11.070272 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-podinfo\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:11.070370 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:11.070295 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-credential-socket\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:11.070370 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:11.070350 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-workload-socket\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:11.070675 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:11.070403 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-data\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:11.070675 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:11.070560 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-workload-certs\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:11.070796 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:11.070768 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-data\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:11.070946 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:11.070925 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8085a328-5dcc-44f9-9b91-6933bbbdd301-istiod-ca-cert\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:11.072920 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:11.072900 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-envoy\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:11.073445 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:11.073423 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-podinfo\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:11.131950 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:11.131917 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-token\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:11.132141 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:11.132047 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7xjg\" (UniqueName: \"kubernetes.io/projected/8085a328-5dcc-44f9-9b91-6933bbbdd301-kube-api-access-x7xjg\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:11.424866 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:11.424780 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:11.920600 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:11.920558 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p"] Oct 30 19:01:11.953547 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:01:11.953516 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8085a328_5dcc_44f9_9b91_6933bbbdd301.slice/crio-d24fc2dcebd2c3e04383881a2a7521307ed744aaff83fd74779069b95b88e6e7 WatchSource:0}: Error finding container d24fc2dcebd2c3e04383881a2a7521307ed744aaff83fd74779069b95b88e6e7: Status 404 returned error can't find the container with id d24fc2dcebd2c3e04383881a2a7521307ed744aaff83fd74779069b95b88e6e7 Oct 30 19:01:11.956384 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:11.956346 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:01:11.956502 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:11.956438 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:01:11.956502 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:11.956485 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:01:12.689188 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:12.689028 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" event={"ID":"8085a328-5dcc-44f9-9b91-6933bbbdd301","Type":"ContainerStarted","Data":"f789b201fd9aab2d1365b951526f56972850d427ce6911fe8c1d69015f341bb9"} Oct 30 19:01:12.689188 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:12.689083 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" event={"ID":"8085a328-5dcc-44f9-9b91-6933bbbdd301","Type":"ContainerStarted","Data":"d24fc2dcebd2c3e04383881a2a7521307ed744aaff83fd74779069b95b88e6e7"} Oct 30 19:01:12.862074 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:12.862009 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" podStartSLOduration=2.8619910280000003 podStartE2EDuration="2.861991028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:12.861602769 +0000 UTC m=+1632.888707727" watchObservedRunningTime="2026-10-30 19:01:12.861991028 +0000 UTC m=+1632.889095988" Oct 30 19:01:13.434156 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:13.430558 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:13.434933 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:13.434902 4531 patch_prober.go:29] interesting pod/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 19:01:13.435150 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:13.435096 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" podUID="8085a328-5dcc-44f9-9b91-6933bbbdd301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 19:01:13.645755 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:13.645685 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj" podUID="9c96011c-d91f-4995-865f-151b4a336c91" containerName="payload-processing" containerID="cri-o://18f79b7dbc639cb88e07e8ccca89461da62e8065bc8acaf6f7961eeff5183706" gracePeriod=25 Oct 30 19:01:13.756587 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:13.756535 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7" podUID="b2dd9790-3a6d-4928-8e56-972f41e9b9c1" containerName="payload-pre-processing" containerID="cri-o://ae9667157b050795cb267c1cf48f93cf9883955e1fa0a35621e5a65181b8b897" gracePeriod=25 Oct 30 19:01:14.199112 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.199086 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj" Oct 30 19:01:14.259502 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.258637 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c96011c-d91f-4995-865f-151b4a336c91-kube-api-access-f8zt8\" (UniqueName: \"kubernetes.io/projected/9c96011c-d91f-4995-865f-151b4a336c91-kube-api-access-f8zt8\") pod \"9c96011c-d91f-4995-865f-151b4a336c91\" (UID: \"9c96011c-d91f-4995-865f-151b4a336c91\") " Oct 30 19:01:14.259502 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.258772 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c96011c-d91f-4995-865f-151b4a336c91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c96011c-d91f-4995-865f-151b4a336c91-plugins-config-volume\") pod \"9c96011c-d91f-4995-865f-151b4a336c91\" (UID: \"9c96011c-d91f-4995-865f-151b4a336c91\") " Oct 30 19:01:14.260627 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.260591 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c96011c-d91f-4995-865f-151b4a336c91-plugins-config-volume" pod "9c96011c-d91f-4995-865f-151b4a336c91" (UID: "9c96011c-d91f-4995-865f-151b4a336c91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:01:14.271679 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.271636 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c96011c-d91f-4995-865f-151b4a336c91-kube-api-access-f8zt8" pod "9c96011c-d91f-4995-865f-151b4a336c91" (UID: "9c96011c-d91f-4995-865f-151b4a336c91"). InnerVolumeSpecName "kube-api-access-f8zt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:14.364218 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.360270 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c96011c-d91f-4995-865f-151b4a336c91-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:01:14.364218 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.360297 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8zt8\" (UniqueName: \"kubernetes.io/projected/9c96011c-d91f-4995-865f-151b4a336c91-kube-api-access-f8zt8\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:01:14.364218 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.361437 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7" Oct 30 19:01:14.441916 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.441573 4531 patch_prober.go:29] interesting pod/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 19:01:14.441916 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.441634 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" podUID="8085a328-5dcc-44f9-9b91-6933bbbdd301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 19:01:14.460935 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.460896 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2dd9790-3a6d-4928-8e56-972f41e9b9c1-kube-api-access-q7c4d\" (UniqueName: \"kubernetes.io/projected/b2dd9790-3a6d-4928-8e56-972f41e9b9c1-kube-api-access-q7c4d\") pod \"b2dd9790-3a6d-4928-8e56-972f41e9b9c1\" (UID: \"b2dd9790-3a6d-4928-8e56-972f41e9b9c1\") " Oct 30 19:01:14.461175 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.461048 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2dd9790-3a6d-4928-8e56-972f41e9b9c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2dd9790-3a6d-4928-8e56-972f41e9b9c1-plugins-config-volume\") pod \"b2dd9790-3a6d-4928-8e56-972f41e9b9c1\" (UID: \"b2dd9790-3a6d-4928-8e56-972f41e9b9c1\") " Oct 30 19:01:14.461464 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.461438 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2dd9790-3a6d-4928-8e56-972f41e9b9c1-plugins-config-volume" pod "b2dd9790-3a6d-4928-8e56-972f41e9b9c1" (UID: "b2dd9790-3a6d-4928-8e56-972f41e9b9c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:01:14.463306 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.463283 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2dd9790-3a6d-4928-8e56-972f41e9b9c1-kube-api-access-q7c4d" pod "b2dd9790-3a6d-4928-8e56-972f41e9b9c1" (UID: "b2dd9790-3a6d-4928-8e56-972f41e9b9c1"). InnerVolumeSpecName "kube-api-access-q7c4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:14.562592 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.562544 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2dd9790-3a6d-4928-8e56-972f41e9b9c1-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:01:14.562592 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.562588 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7c4d\" (UniqueName: \"kubernetes.io/projected/b2dd9790-3a6d-4928-8e56-972f41e9b9c1-kube-api-access-q7c4d\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:01:14.707450 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.707407 4531 generic.go:356] "Generic (PLEG): container finished" podID="b2dd9790-3a6d-4928-8e56-972f41e9b9c1" containerID="ae9667157b050795cb267c1cf48f93cf9883955e1fa0a35621e5a65181b8b897" exitCode=0 Oct 30 19:01:14.707450 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.707456 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7" Oct 30 19:01:14.707728 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.707489 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7" event={"ID":"b2dd9790-3a6d-4928-8e56-972f41e9b9c1","Type":"ContainerDied","Data":"ae9667157b050795cb267c1cf48f93cf9883955e1fa0a35621e5a65181b8b897"} Oct 30 19:01:14.707728 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.707546 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7" event={"ID":"b2dd9790-3a6d-4928-8e56-972f41e9b9c1","Type":"ContainerDied","Data":"d335455278483d7dd650140e8f9ae58891d5a73ea37b86f1aa118a406afec7aa"} Oct 30 19:01:14.707728 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.707570 4531 scope.go:122] "RemoveContainer" containerID="ae9667157b050795cb267c1cf48f93cf9883955e1fa0a35621e5a65181b8b897" Oct 30 19:01:14.709509 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.709477 4531 generic.go:356] "Generic (PLEG): container finished" podID="9c96011c-d91f-4995-865f-151b4a336c91" containerID="18f79b7dbc639cb88e07e8ccca89461da62e8065bc8acaf6f7961eeff5183706" exitCode=0 Oct 30 19:01:14.709634 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.709530 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj" Oct 30 19:01:14.709634 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.709523 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj" event={"ID":"9c96011c-d91f-4995-865f-151b4a336c91","Type":"ContainerDied","Data":"18f79b7dbc639cb88e07e8ccca89461da62e8065bc8acaf6f7961eeff5183706"} Oct 30 19:01:14.709634 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.709629 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj" event={"ID":"9c96011c-d91f-4995-865f-151b4a336c91","Type":"ContainerDied","Data":"b8b5cb54d87ab53e1ea34ac2f8bac159f99bb3be05b85904f3db9cb99a1b2110"} Oct 30 19:01:14.744341 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.744318 4531 scope.go:122] "RemoveContainer" containerID="ae9667157b050795cb267c1cf48f93cf9883955e1fa0a35621e5a65181b8b897" Oct 30 19:01:14.745625 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:01:14.745592 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae9667157b050795cb267c1cf48f93cf9883955e1fa0a35621e5a65181b8b897\": container with ID starting with ae9667157b050795cb267c1cf48f93cf9883955e1fa0a35621e5a65181b8b897 not found: ID does not exist" containerID="ae9667157b050795cb267c1cf48f93cf9883955e1fa0a35621e5a65181b8b897" Oct 30 19:01:14.745777 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.745634 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae9667157b050795cb267c1cf48f93cf9883955e1fa0a35621e5a65181b8b897"} err="failed to get container status \"ae9667157b050795cb267c1cf48f93cf9883955e1fa0a35621e5a65181b8b897\": rpc error: code = NotFound desc = could not find container \"ae9667157b050795cb267c1cf48f93cf9883955e1fa0a35621e5a65181b8b897\": container with ID starting with ae9667157b050795cb267c1cf48f93cf9883955e1fa0a35621e5a65181b8b897 not found: ID does not exist" Oct 30 19:01:14.745777 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.745654 4531 scope.go:122] "RemoveContainer" containerID="18f79b7dbc639cb88e07e8ccca89461da62e8065bc8acaf6f7961eeff5183706" Oct 30 19:01:14.756631 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.756605 4531 scope.go:122] "RemoveContainer" containerID="18f79b7dbc639cb88e07e8ccca89461da62e8065bc8acaf6f7961eeff5183706" Oct 30 19:01:14.756974 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:01:14.756945 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18f79b7dbc639cb88e07e8ccca89461da62e8065bc8acaf6f7961eeff5183706\": container with ID starting with 18f79b7dbc639cb88e07e8ccca89461da62e8065bc8acaf6f7961eeff5183706 not found: ID does not exist" containerID="18f79b7dbc639cb88e07e8ccca89461da62e8065bc8acaf6f7961eeff5183706" Oct 30 19:01:14.757045 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.756983 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f79b7dbc639cb88e07e8ccca89461da62e8065bc8acaf6f7961eeff5183706"} err="failed to get container status \"18f79b7dbc639cb88e07e8ccca89461da62e8065bc8acaf6f7961eeff5183706\": rpc error: code = NotFound desc = could not find container \"18f79b7dbc639cb88e07e8ccca89461da62e8065bc8acaf6f7961eeff5183706\": container with ID starting with 18f79b7dbc639cb88e07e8ccca89461da62e8065bc8acaf6f7961eeff5183706 not found: ID does not exist" Oct 30 19:01:14.905602 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.905568 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj"] Oct 30 19:01:14.909709 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:14.909673 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-85ab6a-57f9cb4976-gfdwj"] Oct 30 19:01:15.041205 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:15.038983 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7"] Oct 30 19:01:15.046088 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:15.046042 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-85ab6a-59fd4dbdff-qvtm7"] Oct 30 19:01:15.442214 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:15.441385 4531 patch_prober.go:29] interesting pod/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 19:01:15.442214 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:15.441454 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" podUID="8085a328-5dcc-44f9-9b91-6933bbbdd301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 19:01:16.099645 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:16.099609 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c96011c-d91f-4995-865f-151b4a336c91" path="/var/lib/kubelet/pods/9c96011c-d91f-4995-865f-151b4a336c91/volumes" Oct 30 19:01:16.100097 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:16.100070 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2dd9790-3a6d-4928-8e56-972f41e9b9c1" path="/var/lib/kubelet/pods/b2dd9790-3a6d-4928-8e56-972f41e9b9c1/volumes" Oct 30 19:01:16.426220 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:16.425639 4531 patch_prober.go:29] interesting pod/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 19:01:16.426220 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:16.425721 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" podUID="8085a328-5dcc-44f9-9b91-6933bbbdd301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 19:01:17.434160 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:17.431579 4531 patch_prober.go:29] interesting pod/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 19:01:17.434160 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:17.431663 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" podUID="8085a328-5dcc-44f9-9b91-6933bbbdd301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 19:01:17.681357 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:17.681306 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10236a-dd79k"] Oct 30 19:01:17.737215 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:17.736538 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-10236a-dd79k"] Oct 30 19:01:18.100578 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:18.100531 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9193de57-f332-4a77-b129-71c9e196714f" path="/var/lib/kubelet/pods/9193de57-f332-4a77-b129-71c9e196714f/volumes" Oct 30 19:01:18.426040 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:18.425931 4531 patch_prober.go:29] interesting pod/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 19:01:18.426040 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:18.426011 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" podUID="8085a328-5dcc-44f9-9b91-6933bbbdd301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 19:01:19.430884 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:19.430852 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:19.431303 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:19.431285 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:19.432162 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:19.432111 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:01:24.248810 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.248770 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd"] Oct 30 19:01:24.249568 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.249546 4531 cpu_manager.go:414] "containerMap: removing container" podUID="b2dd9790-3a6d-4928-8e56-972f41e9b9c1" containerName="payload-pre-processing" Oct 30 19:01:24.249644 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.249571 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2dd9790-3a6d-4928-8e56-972f41e9b9c1" containerName="payload-pre-processing" Oct 30 19:01:24.249644 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.249609 4531 cpu_manager.go:414] "containerMap: removing container" podUID="9c96011c-d91f-4995-865f-151b4a336c91" containerName="payload-processing" Oct 30 19:01:24.249644 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.249616 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c96011c-d91f-4995-865f-151b4a336c91" containerName="payload-processing" Oct 30 19:01:24.249794 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.249783 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c96011c-d91f-4995-865f-151b4a336c91" containerName="payload-processing" Oct 30 19:01:24.249826 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.249808 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2dd9790-3a6d-4928-8e56-972f41e9b9c1" containerName="payload-pre-processing" Oct 30 19:01:24.256422 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.256396 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.351112 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.351070 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-istio-envoy\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.351112 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.351114 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7ea737e-8a33-495c-959e-a037aed04436-istio-token\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.351422 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.351271 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-workload-socket\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.351422 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.351317 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-workload-certs\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.351422 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.351408 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-678fl\" (UniqueName: \"kubernetes.io/projected/b7ea737e-8a33-495c-959e-a037aed04436-kube-api-access-678fl\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.351596 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.351511 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7ea737e-8a33-495c-959e-a037aed04436-istio-podinfo\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.351596 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.351584 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-istio-data\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.351708 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.351663 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-credential-socket\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.351708 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.351700 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7ea737e-8a33-495c-959e-a037aed04436-istiod-ca-cert\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.368182 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.368097 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4ebcd2-openshift-default-dockercfg-2smq2\"" Oct 30 19:01:24.368409 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.368304 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd"] Oct 30 19:01:24.452930 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.452887 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-678fl\" (UniqueName: \"kubernetes.io/projected/b7ea737e-8a33-495c-959e-a037aed04436-kube-api-access-678fl\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.453259 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.453230 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7ea737e-8a33-495c-959e-a037aed04436-istio-podinfo\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.453402 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.453328 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-istio-data\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.453402 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.453383 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-credential-socket\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.453553 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.453429 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7ea737e-8a33-495c-959e-a037aed04436-istiod-ca-cert\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.453631 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.453613 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-istio-envoy\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.453689 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.453651 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7ea737e-8a33-495c-959e-a037aed04436-istio-token\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.453751 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.453735 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-workload-socket\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.453794 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.453761 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-istio-data\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.454105 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.454077 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-credential-socket\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.454221 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.454105 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-workload-certs\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.454221 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.453771 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-workload-certs\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.454714 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.454686 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-workload-socket\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.454959 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.454935 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7ea737e-8a33-495c-959e-a037aed04436-istiod-ca-cert\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.456691 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.456665 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7ea737e-8a33-495c-959e-a037aed04436-istio-podinfo\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.456819 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.456723 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-istio-envoy\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.517989 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.517902 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7ea737e-8a33-495c-959e-a037aed04436-istio-token\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.518889 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.518865 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-678fl\" (UniqueName: \"kubernetes.io/projected/b7ea737e-8a33-495c-959e-a037aed04436-kube-api-access-678fl\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:24.571891 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:24.571850 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:25.013025 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:25.012991 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd"] Oct 30 19:01:25.014072 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:01:25.014032 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7ea737e_8a33_495c_959e_a037aed04436.slice/crio-51d11681020a26022d374cbca8dd47ff2fe75a0a338cdcbba69140175f544ac8 WatchSource:0}: Error finding container 51d11681020a26022d374cbca8dd47ff2fe75a0a338cdcbba69140175f544ac8: Status 404 returned error can't find the container with id 51d11681020a26022d374cbca8dd47ff2fe75a0a338cdcbba69140175f544ac8 Oct 30 19:01:25.017008 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:25.016954 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:01:25.017166 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:25.017054 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:01:25.017166 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:25.017101 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:01:25.843077 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:25.843035 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" event={"ID":"b7ea737e-8a33-495c-959e-a037aed04436","Type":"ContainerStarted","Data":"367a04ed9f6a810e261834b3910bc109784682a4d2eb0b3b633bac5c377088ed"} Oct 30 19:01:25.843592 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:25.843107 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" event={"ID":"b7ea737e-8a33-495c-959e-a037aed04436","Type":"ContainerStarted","Data":"51d11681020a26022d374cbca8dd47ff2fe75a0a338cdcbba69140175f544ac8"} Oct 30 19:01:25.965186 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:25.959199 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6"] Oct 30 19:01:25.965186 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:25.964614 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:25.978667 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:25.978621 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-envoy\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:25.978825 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:25.978708 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-podinfo\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:25.978825 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:25.978763 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-workload-socket\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:25.978825 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:25.978791 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-credential-socket\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:25.978825 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:25.978815 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-data\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:25.979064 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:25.978963 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istiod-ca-cert\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:25.979064 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:25.978984 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q99jv\" (UniqueName: \"kubernetes.io/projected/5a205f4f-e431-45d9-81c4-1f4df8a750ac-kube-api-access-q99jv\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:25.979064 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:25.979020 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-workload-certs\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:25.979325 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:25.979072 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-token\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:25.998405 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:25.998368 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6"] Oct 30 19:01:26.074157 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.050615 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" podStartSLOduration=2.050590984 podStartE2EDuration="2.050590984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:26.048440257 +0000 UTC m=+1646.075545218" watchObservedRunningTime="2026-10-30 19:01:26.050590984 +0000 UTC m=+1646.077695944" Oct 30 19:01:26.266247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.214836 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istiod-ca-cert\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:26.266247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.214890 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q99jv\" (UniqueName: \"kubernetes.io/projected/5a205f4f-e431-45d9-81c4-1f4df8a750ac-kube-api-access-q99jv\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:26.266247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.214941 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-workload-certs\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:26.266247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.215010 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-token\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:26.266247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.215095 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-envoy\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:26.266247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.215176 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-podinfo\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:26.266247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.215222 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-workload-socket\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:26.266247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.215252 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-credential-socket\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:26.266247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.215278 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-data\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:26.266247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.215854 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-data\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:26.266247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.216570 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istiod-ca-cert\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:26.266247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.223237 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-workload-certs\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:26.266247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.230652 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-workload-socket\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:26.266247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.231425 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-credential-socket\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:26.266247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.234696 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-podinfo\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:26.266247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.244510 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-envoy\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:26.279003 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.278959 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6972f60f-nnp8v"] Oct 30 19:01:26.279003 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.279013 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6972f60f-nnp8v"] Oct 30 19:01:26.284844 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.284806 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q99jv\" (UniqueName: \"kubernetes.io/projected/5a205f4f-e431-45d9-81c4-1f4df8a750ac-kube-api-access-q99jv\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:26.292639 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.292606 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-token\") pod \"e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:26.298666 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.298633 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:26.402242 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:01:26.402188 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:01:26.573042 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.572947 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:26.574667 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.574641 4531 patch_prober.go:29] interesting pod/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 19:01:26.574752 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.574692 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" podUID="b7ea737e-8a33-495c-959e-a037aed04436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 19:01:26.788640 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.788607 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6"] Oct 30 19:01:26.789327 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:01:26.789302 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a205f4f_e431_45d9_81c4_1f4df8a750ac.slice/crio-17b350627ca94524a0289e6cdcb3730b03b96f8d9fc50d8815f5064a3a5d4a78 WatchSource:0}: Error finding container 17b350627ca94524a0289e6cdcb3730b03b96f8d9fc50d8815f5064a3a5d4a78: Status 404 returned error can't find the container with id 17b350627ca94524a0289e6cdcb3730b03b96f8d9fc50d8815f5064a3a5d4a78 Oct 30 19:01:26.792258 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.792219 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:01:26.792336 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.792296 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:01:26.792382 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.792332 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:01:26.852642 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.852566 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" event={"ID":"5a205f4f-e431-45d9-81c4-1f4df8a750ac","Type":"ContainerStarted","Data":"17b350627ca94524a0289e6cdcb3730b03b96f8d9fc50d8815f5064a3a5d4a78"} Oct 30 19:01:26.947971 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.947927 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-470093-68ccl"] Oct 30 19:01:26.954669 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:26.952752 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-470093-68ccl" Oct 30 19:01:27.005860 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:27.005646 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:01:27.005860 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:27.005778 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:01:27.005860 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:27.005835 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:01:27.006337 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:27.006296 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:01:27.010510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:27.010470 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-470093-68ccl"] Oct 30 19:01:27.057336 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:27.053845 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c1c263c-35c6-4224-a3f6-dc944720321b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-470093-68ccl\" (UID: \"1c1c263c-35c6-4224-a3f6-dc944720321b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-470093-68ccl" Oct 30 19:01:27.057336 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:27.054064 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzjhs\" (UniqueName: \"kubernetes.io/projected/1c1c263c-35c6-4224-a3f6-dc944720321b-kube-api-access-gzjhs\") pod \"maas-api-revoke-keys-e2e-api-b-470093-68ccl\" (UID: \"1c1c263c-35c6-4224-a3f6-dc944720321b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-470093-68ccl" Oct 30 19:01:27.057336 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:27.054197 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c1c263c-35c6-4224-a3f6-dc944720321b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-470093-68ccl\" (UID: \"1c1c263c-35c6-4224-a3f6-dc944720321b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-470093-68ccl" Oct 30 19:01:27.156195 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:27.156081 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzjhs\" (UniqueName: \"kubernetes.io/projected/1c1c263c-35c6-4224-a3f6-dc944720321b-kube-api-access-gzjhs\") pod \"maas-api-revoke-keys-e2e-api-b-470093-68ccl\" (UID: \"1c1c263c-35c6-4224-a3f6-dc944720321b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-470093-68ccl" Oct 30 19:01:27.156393 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:27.156204 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c1c263c-35c6-4224-a3f6-dc944720321b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-470093-68ccl\" (UID: \"1c1c263c-35c6-4224-a3f6-dc944720321b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-470093-68ccl" Oct 30 19:01:27.156393 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:27.156280 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c1c263c-35c6-4224-a3f6-dc944720321b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-470093-68ccl\" (UID: \"1c1c263c-35c6-4224-a3f6-dc944720321b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-470093-68ccl" Oct 30 19:01:27.156937 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:27.156907 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c1c263c-35c6-4224-a3f6-dc944720321b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-470093-68ccl\" (UID: \"1c1c263c-35c6-4224-a3f6-dc944720321b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-470093-68ccl" Oct 30 19:01:27.159460 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:27.159420 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c1c263c-35c6-4224-a3f6-dc944720321b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-470093-68ccl\" (UID: \"1c1c263c-35c6-4224-a3f6-dc944720321b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-470093-68ccl" Oct 30 19:01:27.237157 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:27.234863 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzjhs\" (UniqueName: \"kubernetes.io/projected/1c1c263c-35c6-4224-a3f6-dc944720321b-kube-api-access-gzjhs\") pod \"maas-api-revoke-keys-e2e-api-b-470093-68ccl\" (UID: \"1c1c263c-35c6-4224-a3f6-dc944720321b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-470093-68ccl" Oct 30 19:01:27.272634 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:27.272596 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-470093-68ccl" Oct 30 19:01:27.573389 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:27.573343 4531 patch_prober.go:29] interesting pod/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 19:01:27.573614 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:27.573408 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" podUID="b7ea737e-8a33-495c-959e-a037aed04436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 19:01:27.693911 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:27.693874 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-470093-68ccl"] Oct 30 19:01:27.694530 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:01:27.694499 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c1c263c_35c6_4224_a3f6_dc944720321b.slice/crio-7d5a857a878e4c5403b7638107e4ff65e2b158e705fb58a9b831b8aa38e87d7a WatchSource:0}: Error finding container 7d5a857a878e4c5403b7638107e4ff65e2b158e705fb58a9b831b8aa38e87d7a: Status 404 returned error can't find the container with id 7d5a857a878e4c5403b7638107e4ff65e2b158e705fb58a9b831b8aa38e87d7a Oct 30 19:01:27.875893 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:27.875810 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" event={"ID":"5a205f4f-e431-45d9-81c4-1f4df8a750ac","Type":"ContainerStarted","Data":"c423c56d2f9d9ea36cbbd4f0e47a6b6517cb7093b72eaaf3ec23f619ceadac2a"} Oct 30 19:01:27.877581 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:27.877548 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-470093-68ccl" event={"ID":"1c1c263c-35c6-4224-a3f6-dc944720321b","Type":"ContainerStarted","Data":"7d5a857a878e4c5403b7638107e4ff65e2b158e705fb58a9b831b8aa38e87d7a"} Oct 30 19:01:28.057175 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:28.028109 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" podStartSLOduration=3.028085503 podStartE2EDuration="3.028085503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:28.013819838 +0000 UTC m=+1648.040924798" watchObservedRunningTime="2026-10-30 19:01:28.028085503 +0000 UTC m=+1648.055190463" Oct 30 19:01:28.101884 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:28.101848 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85de0aae-f351-4133-b2cc-73767de64c96" path="/var/lib/kubelet/pods/85de0aae-f351-4133-b2cc-73767de64c96/volumes" Oct 30 19:01:28.304060 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:28.303859 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:28.304060 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:28.304036 4531 patch_prober.go:29] interesting pod/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 19:01:28.304362 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:28.304094 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" podUID="5a205f4f-e431-45d9-81c4-1f4df8a750ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 19:01:28.572922 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:28.572820 4531 patch_prober.go:29] interesting pod/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 19:01:28.572922 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:28.572886 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" podUID="b7ea737e-8a33-495c-959e-a037aed04436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 19:01:28.886765 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:28.886677 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-470093-68ccl" event={"ID":"1c1c263c-35c6-4224-a3f6-dc944720321b","Type":"ContainerStarted","Data":"2cc141d768e360a0035e711ebbdc4d029da586fbab0657cc9aa24e21d262bb19"} Oct 30 19:01:29.045159 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:29.037115 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-470093-68ccl" podStartSLOduration=3.037091085 podStartE2EDuration="3.037091085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:29.013018654 +0000 UTC m=+1649.040123615" watchObservedRunningTime="2026-10-30 19:01:29.037091085 +0000 UTC m=+1649.064196046" Oct 30 19:01:29.299860 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:29.299824 4531 patch_prober.go:29] interesting pod/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 19:01:29.300102 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:29.299910 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" podUID="5a205f4f-e431-45d9-81c4-1f4df8a750ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 19:01:29.573050 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:29.572954 4531 patch_prober.go:29] interesting pod/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 19:01:29.573050 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:29.573018 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" podUID="b7ea737e-8a33-495c-959e-a037aed04436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 19:01:29.895954 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:29.895858 4531 generic.go:356] "Generic (PLEG): container finished" podID="1c1c263c-35c6-4224-a3f6-dc944720321b" containerID="2cc141d768e360a0035e711ebbdc4d029da586fbab0657cc9aa24e21d262bb19" exitCode=0 Oct 30 19:01:29.895954 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:29.895942 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-470093-68ccl" event={"ID":"1c1c263c-35c6-4224-a3f6-dc944720321b","Type":"ContainerDied","Data":"2cc141d768e360a0035e711ebbdc4d029da586fbab0657cc9aa24e21d262bb19"} Oct 30 19:01:30.301168 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:30.299289 4531 patch_prober.go:29] interesting pod/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 19:01:30.301168 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:30.299360 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" podUID="5a205f4f-e431-45d9-81c4-1f4df8a750ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 19:01:30.589220 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:30.577796 4531 patch_prober.go:29] interesting pod/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 19:01:30.589220 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:30.577863 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" podUID="b7ea737e-8a33-495c-959e-a037aed04436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 19:01:31.302745 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:31.301602 4531 patch_prober.go:29] interesting pod/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 19:01:31.302745 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:31.301664 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" podUID="5a205f4f-e431-45d9-81c4-1f4df8a750ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 19:01:31.516033 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:31.515685 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-470093-68ccl" Oct 30 19:01:31.573599 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:31.573500 4531 patch_prober.go:29] interesting pod/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 30 19:01:31.573599 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:31.573574 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" podUID="b7ea737e-8a33-495c-959e-a037aed04436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 30 19:01:31.691986 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:31.691948 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c1c263c-35c6-4224-a3f6-dc944720321b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c1c263c-35c6-4224-a3f6-dc944720321b-maas-api-xks-ca\") pod \"1c1c263c-35c6-4224-a3f6-dc944720321b\" (UID: \"1c1c263c-35c6-4224-a3f6-dc944720321b\") " Oct 30 19:01:31.692496 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:31.692474 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c1c263c-35c6-4224-a3f6-dc944720321b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c1c263c-35c6-4224-a3f6-dc944720321b-maas-api-openshift-ca\") pod \"1c1c263c-35c6-4224-a3f6-dc944720321b\" (UID: \"1c1c263c-35c6-4224-a3f6-dc944720321b\") " Oct 30 19:01:31.693028 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:31.693008 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c1c263c-35c6-4224-a3f6-dc944720321b-kube-api-access-gzjhs\" (UniqueName: \"kubernetes.io/projected/1c1c263c-35c6-4224-a3f6-dc944720321b-kube-api-access-gzjhs\") pod \"1c1c263c-35c6-4224-a3f6-dc944720321b\" (UID: \"1c1c263c-35c6-4224-a3f6-dc944720321b\") " Oct 30 19:01:31.693196 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:31.693096 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c1c263c-35c6-4224-a3f6-dc944720321b-maas-api-openshift-ca" pod "1c1c263c-35c6-4224-a3f6-dc944720321b" (UID: "1c1c263c-35c6-4224-a3f6-dc944720321b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:01:31.694206 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:31.694092 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c1c263c-35c6-4224-a3f6-dc944720321b-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:01:31.695012 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:31.694982 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c1c263c-35c6-4224-a3f6-dc944720321b-maas-api-xks-ca" pod "1c1c263c-35c6-4224-a3f6-dc944720321b" (UID: "1c1c263c-35c6-4224-a3f6-dc944720321b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:01:31.695683 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:31.695638 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c1c263c-35c6-4224-a3f6-dc944720321b-kube-api-access-gzjhs" pod "1c1c263c-35c6-4224-a3f6-dc944720321b" (UID: "1c1c263c-35c6-4224-a3f6-dc944720321b"). InnerVolumeSpecName "kube-api-access-gzjhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:31.796357 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:31.796302 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c1c263c-35c6-4224-a3f6-dc944720321b-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:01:31.796357 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:31.796330 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzjhs\" (UniqueName: \"kubernetes.io/projected/1c1c263c-35c6-4224-a3f6-dc944720321b-kube-api-access-gzjhs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:01:32.109155 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:32.105301 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-470093-68ccl" Oct 30 19:01:32.109155 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:32.106885 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-470093-68ccl" event={"ID":"1c1c263c-35c6-4224-a3f6-dc944720321b","Type":"ContainerDied","Data":"7d5a857a878e4c5403b7638107e4ff65e2b158e705fb58a9b831b8aa38e87d7a"} Oct 30 19:01:32.109155 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:32.106953 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d5a857a878e4c5403b7638107e4ff65e2b158e705fb58a9b831b8aa38e87d7a" Oct 30 19:01:32.299814 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:32.299775 4531 patch_prober.go:29] interesting pod/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 19:01:32.300006 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:32.299838 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" podUID="5a205f4f-e431-45d9-81c4-1f4df8a750ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 19:01:32.576969 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:32.576936 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:32.577516 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:32.577381 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:32.578285 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:32.578265 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:01:33.299782 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:33.299742 4531 patch_prober.go:29] interesting pod/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 19:01:33.299986 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:33.299803 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" podUID="5a205f4f-e431-45d9-81c4-1f4df8a750ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 19:01:34.305124 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:34.305092 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:34.305649 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:34.305574 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:34.306526 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:34.306506 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:01:44.057165 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:44.057054 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4302db-lb7rb"] Oct 30 19:01:44.107023 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:44.106981 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4302db-lb7rb"] Oct 30 19:01:46.099531 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:46.099497 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b18c67b-7a80-4174-a16f-18219c650cb4" path="/var/lib/kubelet/pods/8b18c67b-7a80-4174-a16f-18219c650cb4/volumes" Oct 30 19:01:46.840415 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:46.840379 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s"] Oct 30 19:01:46.840999 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:46.840982 4531 cpu_manager.go:414] "containerMap: removing container" podUID="1c1c263c-35c6-4224-a3f6-dc944720321b" containerName="revoke-keys" Oct 30 19:01:46.841067 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:46.841000 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c1c263c-35c6-4224-a3f6-dc944720321b" containerName="revoke-keys" Oct 30 19:01:46.841110 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:46.841103 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c1c263c-35c6-4224-a3f6-dc944720321b" containerName="revoke-keys" Oct 30 19:01:46.845981 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:46.845961 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s" Oct 30 19:01:46.895545 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:46.895511 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6b35e72-807a-48b1-a764-9698ffe71dc2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s\" (UID: \"e6b35e72-807a-48b1-a764-9698ffe71dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s" Oct 30 19:01:46.895762 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:46.895715 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6b35e72-807a-48b1-a764-9698ffe71dc2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s\" (UID: \"e6b35e72-807a-48b1-a764-9698ffe71dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s" Oct 30 19:01:46.895832 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:46.895811 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bszsm\" (UniqueName: \"kubernetes.io/projected/e6b35e72-807a-48b1-a764-9698ffe71dc2-kube-api-access-bszsm\") pod \"maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s\" (UID: \"e6b35e72-807a-48b1-a764-9698ffe71dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s" Oct 30 19:01:46.947330 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:46.947295 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:01:46.947546 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:46.947295 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:01:46.947546 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:46.947301 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:01:46.947546 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:46.947453 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:01:46.951369 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:46.951337 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s"] Oct 30 19:01:46.997653 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:46.997616 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6b35e72-807a-48b1-a764-9698ffe71dc2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s\" (UID: \"e6b35e72-807a-48b1-a764-9698ffe71dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s" Oct 30 19:01:46.997846 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:46.997826 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6b35e72-807a-48b1-a764-9698ffe71dc2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s\" (UID: \"e6b35e72-807a-48b1-a764-9698ffe71dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s" Oct 30 19:01:46.997938 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:46.997923 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bszsm\" (UniqueName: \"kubernetes.io/projected/e6b35e72-807a-48b1-a764-9698ffe71dc2-kube-api-access-bszsm\") pod \"maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s\" (UID: \"e6b35e72-807a-48b1-a764-9698ffe71dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s" Oct 30 19:01:46.998506 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:46.998474 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6b35e72-807a-48b1-a764-9698ffe71dc2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s\" (UID: \"e6b35e72-807a-48b1-a764-9698ffe71dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s" Oct 30 19:01:47.000255 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:47.000230 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6b35e72-807a-48b1-a764-9698ffe71dc2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s\" (UID: \"e6b35e72-807a-48b1-a764-9698ffe71dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s" Oct 30 19:01:47.062403 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:47.062369 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bszsm\" (UniqueName: \"kubernetes.io/projected/e6b35e72-807a-48b1-a764-9698ffe71dc2-kube-api-access-bszsm\") pod \"maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s\" (UID: \"e6b35e72-807a-48b1-a764-9698ffe71dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s" Oct 30 19:01:47.164824 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:47.164747 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s" Oct 30 19:01:47.549905 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:47.549874 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s"] Oct 30 19:01:47.550232 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:01:47.550211 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b35e72_807a_48b1_a764_9698ffe71dc2.slice/crio-9b12388d2f7ba4cbbc85f9c6f7d42e4b426045c2de4038bef07dc54d04828dd1 WatchSource:0}: Error finding container 9b12388d2f7ba4cbbc85f9c6f7d42e4b426045c2de4038bef07dc54d04828dd1: Status 404 returned error can't find the container with id 9b12388d2f7ba4cbbc85f9c6f7d42e4b426045c2de4038bef07dc54d04828dd1 Oct 30 19:01:48.222532 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:48.222440 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5"] Oct 30 19:01:48.229063 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:48.229019 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s" event={"ID":"e6b35e72-807a-48b1-a764-9698ffe71dc2","Type":"ContainerStarted","Data":"19a2977eaa3897cc33c02c364d63cd2ded5ce45f10074aef1bce02c2213c18bd"} Oct 30 19:01:48.229238 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:48.229083 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s" event={"ID":"e6b35e72-807a-48b1-a764-9698ffe71dc2","Type":"ContainerStarted","Data":"9b12388d2f7ba4cbbc85f9c6f7d42e4b426045c2de4038bef07dc54d04828dd1"} Oct 30 19:01:48.456668 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:48.456605 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s" podStartSLOduration=2.4565850129999998 podStartE2EDuration="2.456585013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:48.455976756 +0000 UTC m=+1668.483081714" watchObservedRunningTime="2026-10-30 19:01:48.456585013 +0000 UTC m=+1668.483689977" Oct 30 19:01:49.239709 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:49.239675 4531 generic.go:356] "Generic (PLEG): container finished" podID="e6b35e72-807a-48b1-a764-9698ffe71dc2" containerID="19a2977eaa3897cc33c02c364d63cd2ded5ce45f10074aef1bce02c2213c18bd" exitCode=0 Oct 30 19:01:49.240179 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:49.239752 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s" event={"ID":"e6b35e72-807a-48b1-a764-9698ffe71dc2","Type":"ContainerDied","Data":"19a2977eaa3897cc33c02c364d63cd2ded5ce45f10074aef1bce02c2213c18bd"} Oct 30 19:01:50.415484 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:50.415454 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s" Oct 30 19:01:50.595994 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:50.595875 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6b35e72-807a-48b1-a764-9698ffe71dc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6b35e72-807a-48b1-a764-9698ffe71dc2-maas-api-xks-ca\") pod \"e6b35e72-807a-48b1-a764-9698ffe71dc2\" (UID: \"e6b35e72-807a-48b1-a764-9698ffe71dc2\") " Oct 30 19:01:50.595994 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:50.595966 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6b35e72-807a-48b1-a764-9698ffe71dc2-kube-api-access-bszsm\" (UniqueName: \"kubernetes.io/projected/e6b35e72-807a-48b1-a764-9698ffe71dc2-kube-api-access-bszsm\") pod \"e6b35e72-807a-48b1-a764-9698ffe71dc2\" (UID: \"e6b35e72-807a-48b1-a764-9698ffe71dc2\") " Oct 30 19:01:50.596266 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:50.596079 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6b35e72-807a-48b1-a764-9698ffe71dc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6b35e72-807a-48b1-a764-9698ffe71dc2-maas-api-openshift-ca\") pod \"e6b35e72-807a-48b1-a764-9698ffe71dc2\" (UID: \"e6b35e72-807a-48b1-a764-9698ffe71dc2\") " Oct 30 19:01:50.596623 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:50.596595 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6b35e72-807a-48b1-a764-9698ffe71dc2-maas-api-openshift-ca" pod "e6b35e72-807a-48b1-a764-9698ffe71dc2" (UID: "e6b35e72-807a-48b1-a764-9698ffe71dc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:01:50.596822 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:50.596802 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6b35e72-807a-48b1-a764-9698ffe71dc2-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:01:50.598964 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:50.598935 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6b35e72-807a-48b1-a764-9698ffe71dc2-maas-api-xks-ca" pod "e6b35e72-807a-48b1-a764-9698ffe71dc2" (UID: "e6b35e72-807a-48b1-a764-9698ffe71dc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:01:50.598964 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:50.598950 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b35e72-807a-48b1-a764-9698ffe71dc2-kube-api-access-bszsm" pod "e6b35e72-807a-48b1-a764-9698ffe71dc2" (UID: "e6b35e72-807a-48b1-a764-9698ffe71dc2"). InnerVolumeSpecName "kube-api-access-bszsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:50.698000 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:50.697964 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6b35e72-807a-48b1-a764-9698ffe71dc2-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:01:50.698000 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:50.698000 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bszsm\" (UniqueName: \"kubernetes.io/projected/e6b35e72-807a-48b1-a764-9698ffe71dc2-kube-api-access-bszsm\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:01:51.239409 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:51.239367 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8"] Oct 30 19:01:51.240029 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:51.240012 4531 cpu_manager.go:414] "containerMap: removing container" podUID="e6b35e72-807a-48b1-a764-9698ffe71dc2" containerName="revoke-keys" Oct 30 19:01:51.240088 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:51.240032 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6b35e72-807a-48b1-a764-9698ffe71dc2" containerName="revoke-keys" Oct 30 19:01:51.240180 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:51.240170 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6b35e72-807a-48b1-a764-9698ffe71dc2" containerName="revoke-keys" Oct 30 19:01:51.243732 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:51.243710 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8" Oct 30 19:01:51.260880 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:51.260840 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s" Oct 30 19:01:51.261038 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:51.260871 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s" event={"ID":"e6b35e72-807a-48b1-a764-9698ffe71dc2","Type":"ContainerDied","Data":"9b12388d2f7ba4cbbc85f9c6f7d42e4b426045c2de4038bef07dc54d04828dd1"} Oct 30 19:01:51.261038 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:51.260944 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b12388d2f7ba4cbbc85f9c6f7d42e4b426045c2de4038bef07dc54d04828dd1" Oct 30 19:01:51.302291 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:51.302254 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8"] Oct 30 19:01:51.406953 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:51.406898 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl9nw\" (UniqueName: \"kubernetes.io/projected/f07706fa-4548-44ac-8b1f-bcad8a391dfd-kube-api-access-sl9nw\") pod \"maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8\" (UID: \"f07706fa-4548-44ac-8b1f-bcad8a391dfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8" Oct 30 19:01:51.407211 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:51.407008 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07706fa-4548-44ac-8b1f-bcad8a391dfd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8\" (UID: \"f07706fa-4548-44ac-8b1f-bcad8a391dfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8" Oct 30 19:01:51.407211 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:51.407115 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07706fa-4548-44ac-8b1f-bcad8a391dfd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8\" (UID: \"f07706fa-4548-44ac-8b1f-bcad8a391dfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8" Oct 30 19:01:51.508945 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:51.508899 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl9nw\" (UniqueName: \"kubernetes.io/projected/f07706fa-4548-44ac-8b1f-bcad8a391dfd-kube-api-access-sl9nw\") pod \"maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8\" (UID: \"f07706fa-4548-44ac-8b1f-bcad8a391dfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8" Oct 30 19:01:51.509484 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:51.508995 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07706fa-4548-44ac-8b1f-bcad8a391dfd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8\" (UID: \"f07706fa-4548-44ac-8b1f-bcad8a391dfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8" Oct 30 19:01:51.509484 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:51.509145 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07706fa-4548-44ac-8b1f-bcad8a391dfd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8\" (UID: \"f07706fa-4548-44ac-8b1f-bcad8a391dfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8" Oct 30 19:01:51.509863 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:51.509838 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07706fa-4548-44ac-8b1f-bcad8a391dfd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8\" (UID: \"f07706fa-4548-44ac-8b1f-bcad8a391dfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8" Oct 30 19:01:51.511703 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:51.511677 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07706fa-4548-44ac-8b1f-bcad8a391dfd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8\" (UID: \"f07706fa-4548-44ac-8b1f-bcad8a391dfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8" Oct 30 19:01:51.571275 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:51.571236 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl9nw\" (UniqueName: \"kubernetes.io/projected/f07706fa-4548-44ac-8b1f-bcad8a391dfd-kube-api-access-sl9nw\") pod \"maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8\" (UID: \"f07706fa-4548-44ac-8b1f-bcad8a391dfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8" Oct 30 19:01:51.861411 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:51.861319 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8" Oct 30 19:01:52.246978 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:52.246944 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8"] Oct 30 19:01:52.247824 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:01:52.247800 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07706fa_4548_44ac_8b1f_bcad8a391dfd.slice/crio-f6384658e68c90774582a3b4ef879e0155c454844224dc4b1521cb1cf2f8c872 WatchSource:0}: Error finding container f6384658e68c90774582a3b4ef879e0155c454844224dc4b1521cb1cf2f8c872: Status 404 returned error can't find the container with id f6384658e68c90774582a3b4ef879e0155c454844224dc4b1521cb1cf2f8c872 Oct 30 19:01:52.271352 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:52.271312 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8" event={"ID":"f07706fa-4548-44ac-8b1f-bcad8a391dfd","Type":"ContainerStarted","Data":"f6384658e68c90774582a3b4ef879e0155c454844224dc4b1521cb1cf2f8c872"} Oct 30 19:01:53.223316 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:53.223226 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5" podUID="8974a32e-1a13-435a-b4cc-1948881c3532" containerName="payload-pre-processing" containerID="cri-o://27476dec085c2c4c97a6fd27462fd641861bbd5c084164965e37595d58f2d1fc" gracePeriod=25 Oct 30 19:01:53.280373 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:53.280343 4531 generic.go:356] "Generic (PLEG): container finished" podID="f07706fa-4548-44ac-8b1f-bcad8a391dfd" containerID="d6106021d164c92114c1044ab15c9f99887e3e32747ab0d5fcba5b4b53f3753e" exitCode=0 Oct 30 19:01:53.280521 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:53.280419 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8" event={"ID":"f07706fa-4548-44ac-8b1f-bcad8a391dfd","Type":"ContainerDied","Data":"d6106021d164c92114c1044ab15c9f99887e3e32747ab0d5fcba5b4b53f3753e"} Oct 30 19:01:53.522616 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:53.522584 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5" Oct 30 19:01:53.534724 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:53.534695 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8974a32e-1a13-435a-b4cc-1948881c3532-kube-api-access-jcbp9\" (UniqueName: \"kubernetes.io/projected/8974a32e-1a13-435a-b4cc-1948881c3532-kube-api-access-jcbp9\") pod \"8974a32e-1a13-435a-b4cc-1948881c3532\" (UID: \"8974a32e-1a13-435a-b4cc-1948881c3532\") " Oct 30 19:01:53.534981 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:53.534936 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8974a32e-1a13-435a-b4cc-1948881c3532-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8974a32e-1a13-435a-b4cc-1948881c3532-plugins-config-volume\") pod \"8974a32e-1a13-435a-b4cc-1948881c3532\" (UID: \"8974a32e-1a13-435a-b4cc-1948881c3532\") " Oct 30 19:01:53.535354 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:53.535331 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8974a32e-1a13-435a-b4cc-1948881c3532-plugins-config-volume" pod "8974a32e-1a13-435a-b4cc-1948881c3532" (UID: "8974a32e-1a13-435a-b4cc-1948881c3532"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:01:53.535513 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:53.535496 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8974a32e-1a13-435a-b4cc-1948881c3532-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:01:53.537118 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:53.537091 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8974a32e-1a13-435a-b4cc-1948881c3532-kube-api-access-jcbp9" pod "8974a32e-1a13-435a-b4cc-1948881c3532" (UID: "8974a32e-1a13-435a-b4cc-1948881c3532"). InnerVolumeSpecName "kube-api-access-jcbp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:53.637287 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:53.637244 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcbp9\" (UniqueName: \"kubernetes.io/projected/8974a32e-1a13-435a-b4cc-1948881c3532-kube-api-access-jcbp9\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:01:54.289766 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:54.289727 4531 generic.go:356] "Generic (PLEG): container finished" podID="8974a32e-1a13-435a-b4cc-1948881c3532" containerID="27476dec085c2c4c97a6fd27462fd641861bbd5c084164965e37595d58f2d1fc" exitCode=0 Oct 30 19:01:54.290301 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:54.289796 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5" Oct 30 19:01:54.290301 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:54.289815 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5" event={"ID":"8974a32e-1a13-435a-b4cc-1948881c3532","Type":"ContainerDied","Data":"27476dec085c2c4c97a6fd27462fd641861bbd5c084164965e37595d58f2d1fc"} Oct 30 19:01:54.290301 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:54.289874 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5" event={"ID":"8974a32e-1a13-435a-b4cc-1948881c3532","Type":"ContainerDied","Data":"6869226c0c56434b8ffafa23d3ed12b3f5fed1af4648f1354462cdf14e026e44"} Oct 30 19:01:54.290301 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:54.289918 4531 scope.go:122] "RemoveContainer" containerID="27476dec085c2c4c97a6fd27462fd641861bbd5c084164965e37595d58f2d1fc" Oct 30 19:01:54.300368 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:54.300343 4531 scope.go:122] "RemoveContainer" containerID="27476dec085c2c4c97a6fd27462fd641861bbd5c084164965e37595d58f2d1fc" Oct 30 19:01:54.300703 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:01:54.300673 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27476dec085c2c4c97a6fd27462fd641861bbd5c084164965e37595d58f2d1fc\": container with ID starting with 27476dec085c2c4c97a6fd27462fd641861bbd5c084164965e37595d58f2d1fc not found: ID does not exist" containerID="27476dec085c2c4c97a6fd27462fd641861bbd5c084164965e37595d58f2d1fc" Oct 30 19:01:54.300815 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:54.300709 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27476dec085c2c4c97a6fd27462fd641861bbd5c084164965e37595d58f2d1fc"} err="failed to get container status \"27476dec085c2c4c97a6fd27462fd641861bbd5c084164965e37595d58f2d1fc\": rpc error: code = NotFound desc = could not find container \"27476dec085c2c4c97a6fd27462fd641861bbd5c084164965e37595d58f2d1fc\": container with ID starting with 27476dec085c2c4c97a6fd27462fd641861bbd5c084164965e37595d58f2d1fc not found: ID does not exist" Oct 30 19:01:54.456528 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:54.456504 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5"] Oct 30 19:01:54.457794 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:54.457776 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8" Oct 30 19:01:54.506066 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:54.506036 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-470093-c86dc647-fxvf5"] Oct 30 19:01:54.554733 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:54.554651 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f07706fa-4548-44ac-8b1f-bcad8a391dfd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07706fa-4548-44ac-8b1f-bcad8a391dfd-maas-api-xks-ca\") pod \"f07706fa-4548-44ac-8b1f-bcad8a391dfd\" (UID: \"f07706fa-4548-44ac-8b1f-bcad8a391dfd\") " Oct 30 19:01:54.554866 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:54.554774 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f07706fa-4548-44ac-8b1f-bcad8a391dfd-kube-api-access-sl9nw\" (UniqueName: \"kubernetes.io/projected/f07706fa-4548-44ac-8b1f-bcad8a391dfd-kube-api-access-sl9nw\") pod \"f07706fa-4548-44ac-8b1f-bcad8a391dfd\" (UID: \"f07706fa-4548-44ac-8b1f-bcad8a391dfd\") " Oct 30 19:01:54.554866 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:54.554852 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f07706fa-4548-44ac-8b1f-bcad8a391dfd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07706fa-4548-44ac-8b1f-bcad8a391dfd-maas-api-openshift-ca\") pod \"f07706fa-4548-44ac-8b1f-bcad8a391dfd\" (UID: \"f07706fa-4548-44ac-8b1f-bcad8a391dfd\") " Oct 30 19:01:54.555281 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:54.555246 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f07706fa-4548-44ac-8b1f-bcad8a391dfd-maas-api-openshift-ca" pod "f07706fa-4548-44ac-8b1f-bcad8a391dfd" (UID: "f07706fa-4548-44ac-8b1f-bcad8a391dfd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:01:54.555382 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:54.555365 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07706fa-4548-44ac-8b1f-bcad8a391dfd-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:01:54.556886 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:54.556863 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f07706fa-4548-44ac-8b1f-bcad8a391dfd-maas-api-xks-ca" pod "f07706fa-4548-44ac-8b1f-bcad8a391dfd" (UID: "f07706fa-4548-44ac-8b1f-bcad8a391dfd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:01:54.556945 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:54.556926 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07706fa-4548-44ac-8b1f-bcad8a391dfd-kube-api-access-sl9nw" pod "f07706fa-4548-44ac-8b1f-bcad8a391dfd" (UID: "f07706fa-4548-44ac-8b1f-bcad8a391dfd"). InnerVolumeSpecName "kube-api-access-sl9nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:54.657205 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:54.657169 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07706fa-4548-44ac-8b1f-bcad8a391dfd-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:01:54.657205 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:54.657204 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl9nw\" (UniqueName: \"kubernetes.io/projected/f07706fa-4548-44ac-8b1f-bcad8a391dfd-kube-api-access-sl9nw\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:01:55.052146 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:55.052080 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf555aa7-p47r8"] Oct 30 19:01:55.066167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:55.066100 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf555aa7-p47r8"] Oct 30 19:01:55.297456 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:55.297430 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8" Oct 30 19:01:55.297930 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:55.297472 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8" event={"ID":"f07706fa-4548-44ac-8b1f-bcad8a391dfd","Type":"ContainerDied","Data":"f6384658e68c90774582a3b4ef879e0155c454844224dc4b1521cb1cf2f8c872"} Oct 30 19:01:55.297930 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:55.297528 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6384658e68c90774582a3b4ef879e0155c454844224dc4b1521cb1cf2f8c872" Oct 30 19:01:56.100481 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:56.100448 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87f754d5-f500-47c8-9026-07737a66b6ea" path="/var/lib/kubelet/pods/87f754d5-f500-47c8-9026-07737a66b6ea/volumes" Oct 30 19:01:56.100883 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:56.100871 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8974a32e-1a13-435a-b4cc-1948881c3532" path="/var/lib/kubelet/pods/8974a32e-1a13-435a-b4cc-1948881c3532/volumes" Oct 30 19:01:56.398444 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:01:56.398357 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:01:58.346500 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:58.346469 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j"] Oct 30 19:01:58.451270 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:01:58.451233 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx"] Oct 30 19:02:01.172060 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:01.172021 4531 scope.go:122] "RemoveContainer" containerID="ed98cd01a1f723fbf1838c96276a843441baf00f7e0308f0aa9c6eb73824a249" Oct 30 19:02:01.182646 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:01.182619 4531 scope.go:122] "RemoveContainer" containerID="c30cbde9692305c53435962479be92fa3329f6fe40981cd770e586820bbf75c3" Oct 30 19:02:02.817501 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:02.817464 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn"] Oct 30 19:02:02.934053 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:02.933991 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t"] Oct 30 19:02:03.350933 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:03.350871 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j" podUID="a58321dd-1bd5-45e5-89d1-1a9a60aaadf9" containerName="payload-processing" containerID="cri-o://75229c19a319013e42870d3c90b5ffc2138279a4819203c206d3eb8712fc2f31" gracePeriod=25 Oct 30 19:02:03.451699 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:03.451629 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx" podUID="a860f1c1-b883-44ba-bc53-e20c606ae368" containerName="payload-pre-processing" containerID="cri-o://df4f7b6ecc720ac10e78d5a18b2c10e44cdc45f746c5d370c37d37ff88ab1716" gracePeriod=25 Oct 30 19:02:03.775675 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:03.775644 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j" Oct 30 19:02:03.796232 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:03.796199 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx" Oct 30 19:02:03.899733 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:03.899642 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a58321dd-1bd5-45e5-89d1-1a9a60aaadf9-kube-api-access-v7hts\" (UniqueName: \"kubernetes.io/projected/a58321dd-1bd5-45e5-89d1-1a9a60aaadf9-kube-api-access-v7hts\") pod \"a58321dd-1bd5-45e5-89d1-1a9a60aaadf9\" (UID: \"a58321dd-1bd5-45e5-89d1-1a9a60aaadf9\") " Oct 30 19:02:03.900194 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:03.899741 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a58321dd-1bd5-45e5-89d1-1a9a60aaadf9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a58321dd-1bd5-45e5-89d1-1a9a60aaadf9-plugins-config-volume\") pod \"a58321dd-1bd5-45e5-89d1-1a9a60aaadf9\" (UID: \"a58321dd-1bd5-45e5-89d1-1a9a60aaadf9\") " Oct 30 19:02:03.900194 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:03.899905 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a860f1c1-b883-44ba-bc53-e20c606ae368-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a860f1c1-b883-44ba-bc53-e20c606ae368-plugins-config-volume\") pod \"a860f1c1-b883-44ba-bc53-e20c606ae368\" (UID: \"a860f1c1-b883-44ba-bc53-e20c606ae368\") " Oct 30 19:02:03.900194 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:03.899942 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a860f1c1-b883-44ba-bc53-e20c606ae368-kube-api-access-4dt4b\" (UniqueName: \"kubernetes.io/projected/a860f1c1-b883-44ba-bc53-e20c606ae368-kube-api-access-4dt4b\") pod \"a860f1c1-b883-44ba-bc53-e20c606ae368\" (UID: \"a860f1c1-b883-44ba-bc53-e20c606ae368\") " Oct 30 19:02:03.900391 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:03.900197 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a58321dd-1bd5-45e5-89d1-1a9a60aaadf9-plugins-config-volume" pod "a58321dd-1bd5-45e5-89d1-1a9a60aaadf9" (UID: "a58321dd-1bd5-45e5-89d1-1a9a60aaadf9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:02:03.900391 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:03.900324 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a860f1c1-b883-44ba-bc53-e20c606ae368-plugins-config-volume" pod "a860f1c1-b883-44ba-bc53-e20c606ae368" (UID: "a860f1c1-b883-44ba-bc53-e20c606ae368"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:02:03.900735 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:03.900689 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a58321dd-1bd5-45e5-89d1-1a9a60aaadf9-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:03.900735 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:03.900713 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a860f1c1-b883-44ba-bc53-e20c606ae368-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:03.902522 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:03.902490 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a860f1c1-b883-44ba-bc53-e20c606ae368-kube-api-access-4dt4b" pod "a860f1c1-b883-44ba-bc53-e20c606ae368" (UID: "a860f1c1-b883-44ba-bc53-e20c606ae368"). InnerVolumeSpecName "kube-api-access-4dt4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:02:03.902646 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:03.902613 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a58321dd-1bd5-45e5-89d1-1a9a60aaadf9-kube-api-access-v7hts" pod "a58321dd-1bd5-45e5-89d1-1a9a60aaadf9" (UID: "a58321dd-1bd5-45e5-89d1-1a9a60aaadf9"). InnerVolumeSpecName "kube-api-access-v7hts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:02:04.001870 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:04.001834 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dt4b\" (UniqueName: \"kubernetes.io/projected/a860f1c1-b883-44ba-bc53-e20c606ae368-kube-api-access-4dt4b\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:04.001870 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:04.001872 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7hts\" (UniqueName: \"kubernetes.io/projected/a58321dd-1bd5-45e5-89d1-1a9a60aaadf9-kube-api-access-v7hts\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:04.430153 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:04.425526 4531 generic.go:356] "Generic (PLEG): container finished" podID="a860f1c1-b883-44ba-bc53-e20c606ae368" containerID="df4f7b6ecc720ac10e78d5a18b2c10e44cdc45f746c5d370c37d37ff88ab1716" exitCode=0 Oct 30 19:02:04.430153 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:04.425971 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx" Oct 30 19:02:04.434218 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:04.431444 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx" event={"ID":"a860f1c1-b883-44ba-bc53-e20c606ae368","Type":"ContainerDied","Data":"df4f7b6ecc720ac10e78d5a18b2c10e44cdc45f746c5d370c37d37ff88ab1716"} Oct 30 19:02:04.434218 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:04.431518 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx" event={"ID":"a860f1c1-b883-44ba-bc53-e20c606ae368","Type":"ContainerDied","Data":"afe58f875259ae2998a1c7cc2c6b75f7a4890205b86c49369f40d028faa6281d"} Oct 30 19:02:04.434218 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:04.431548 4531 scope.go:122] "RemoveContainer" containerID="df4f7b6ecc720ac10e78d5a18b2c10e44cdc45f746c5d370c37d37ff88ab1716" Oct 30 19:02:04.439633 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:04.439441 4531 generic.go:356] "Generic (PLEG): container finished" podID="a58321dd-1bd5-45e5-89d1-1a9a60aaadf9" containerID="75229c19a319013e42870d3c90b5ffc2138279a4819203c206d3eb8712fc2f31" exitCode=0 Oct 30 19:02:04.439836 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:04.439773 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j" Oct 30 19:02:04.443026 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:04.442999 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j" event={"ID":"a58321dd-1bd5-45e5-89d1-1a9a60aaadf9","Type":"ContainerDied","Data":"75229c19a319013e42870d3c90b5ffc2138279a4819203c206d3eb8712fc2f31"} Oct 30 19:02:04.443229 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:04.443197 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j" event={"ID":"a58321dd-1bd5-45e5-89d1-1a9a60aaadf9","Type":"ContainerDied","Data":"618fc7f512602fabfa8efd748e721cbea0fc84c7ea2cd5a11f935ba1829652a5"} Oct 30 19:02:04.457110 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:04.457080 4531 scope.go:122] "RemoveContainer" containerID="df4f7b6ecc720ac10e78d5a18b2c10e44cdc45f746c5d370c37d37ff88ab1716" Oct 30 19:02:04.457588 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:02:04.457561 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df4f7b6ecc720ac10e78d5a18b2c10e44cdc45f746c5d370c37d37ff88ab1716\": container with ID starting with df4f7b6ecc720ac10e78d5a18b2c10e44cdc45f746c5d370c37d37ff88ab1716 not found: ID does not exist" containerID="df4f7b6ecc720ac10e78d5a18b2c10e44cdc45f746c5d370c37d37ff88ab1716" Oct 30 19:02:04.457690 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:04.457602 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df4f7b6ecc720ac10e78d5a18b2c10e44cdc45f746c5d370c37d37ff88ab1716"} err="failed to get container status \"df4f7b6ecc720ac10e78d5a18b2c10e44cdc45f746c5d370c37d37ff88ab1716\": rpc error: code = NotFound desc = could not find container \"df4f7b6ecc720ac10e78d5a18b2c10e44cdc45f746c5d370c37d37ff88ab1716\": container with ID starting with df4f7b6ecc720ac10e78d5a18b2c10e44cdc45f746c5d370c37d37ff88ab1716 not found: ID does not exist" Oct 30 19:02:04.457690 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:04.457626 4531 scope.go:122] "RemoveContainer" containerID="75229c19a319013e42870d3c90b5ffc2138279a4819203c206d3eb8712fc2f31" Oct 30 19:02:04.472909 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:04.472880 4531 scope.go:122] "RemoveContainer" containerID="75229c19a319013e42870d3c90b5ffc2138279a4819203c206d3eb8712fc2f31" Oct 30 19:02:04.473373 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:02:04.473336 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75229c19a319013e42870d3c90b5ffc2138279a4819203c206d3eb8712fc2f31\": container with ID starting with 75229c19a319013e42870d3c90b5ffc2138279a4819203c206d3eb8712fc2f31 not found: ID does not exist" containerID="75229c19a319013e42870d3c90b5ffc2138279a4819203c206d3eb8712fc2f31" Oct 30 19:02:04.473495 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:04.473389 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75229c19a319013e42870d3c90b5ffc2138279a4819203c206d3eb8712fc2f31"} err="failed to get container status \"75229c19a319013e42870d3c90b5ffc2138279a4819203c206d3eb8712fc2f31\": rpc error: code = NotFound desc = could not find container \"75229c19a319013e42870d3c90b5ffc2138279a4819203c206d3eb8712fc2f31\": container with ID starting with 75229c19a319013e42870d3c90b5ffc2138279a4819203c206d3eb8712fc2f31 not found: ID does not exist" Oct 30 19:02:04.623555 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:04.623517 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx"] Oct 30 19:02:04.628825 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:04.628775 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-83d5bd-66b84cb5d7-hz2jx"] Oct 30 19:02:04.722432 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:04.722337 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j"] Oct 30 19:02:04.730597 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:04.730548 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-83d5bd-748bd86ffd-j8r9j"] Oct 30 19:02:06.099122 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:06.099089 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a58321dd-1bd5-45e5-89d1-1a9a60aaadf9" path="/var/lib/kubelet/pods/a58321dd-1bd5-45e5-89d1-1a9a60aaadf9/volumes" Oct 30 19:02:06.099563 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:06.099547 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a860f1c1-b883-44ba-bc53-e20c606ae368" path="/var/lib/kubelet/pods/a860f1c1-b883-44ba-bc53-e20c606ae368/volumes" Oct 30 19:02:07.818167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:07.818086 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn" podUID="c6812606-a602-4a6d-b226-4be22c94d655" containerName="payload-processing" containerID="cri-o://5a31c070b8016132f34144a38afb9ac313dcff0f74d3dc8dfa5ca56692c74ab7" gracePeriod=25 Oct 30 19:02:07.934975 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:07.934919 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t" podUID="40617548-d684-4609-9c1e-37b58c1e4bfb" containerName="payload-pre-processing" containerID="cri-o://234635f6e6d8cc7e59acc6812c799418644a264a39159206dc5a07d8c2fd9879" gracePeriod=25 Oct 30 19:02:08.095465 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.095436 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn" Oct 30 19:02:08.173954 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.173897 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6812606-a602-4a6d-b226-4be22c94d655-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6812606-a602-4a6d-b226-4be22c94d655-plugins-config-volume\") pod \"c6812606-a602-4a6d-b226-4be22c94d655\" (UID: \"c6812606-a602-4a6d-b226-4be22c94d655\") " Oct 30 19:02:08.174219 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.174174 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6812606-a602-4a6d-b226-4be22c94d655-kube-api-access-b4rlq\" (UniqueName: \"kubernetes.io/projected/c6812606-a602-4a6d-b226-4be22c94d655-kube-api-access-b4rlq\") pod \"c6812606-a602-4a6d-b226-4be22c94d655\" (UID: \"c6812606-a602-4a6d-b226-4be22c94d655\") " Oct 30 19:02:08.174466 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.174435 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6812606-a602-4a6d-b226-4be22c94d655-plugins-config-volume" pod "c6812606-a602-4a6d-b226-4be22c94d655" (UID: "c6812606-a602-4a6d-b226-4be22c94d655"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:02:08.174798 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.174772 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6812606-a602-4a6d-b226-4be22c94d655-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:08.177001 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.176974 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6812606-a602-4a6d-b226-4be22c94d655-kube-api-access-b4rlq" pod "c6812606-a602-4a6d-b226-4be22c94d655" (UID: "c6812606-a602-4a6d-b226-4be22c94d655"). InnerVolumeSpecName "kube-api-access-b4rlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:02:08.211219 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.211189 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t" Oct 30 19:02:08.275363 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.275328 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40617548-d684-4609-9c1e-37b58c1e4bfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40617548-d684-4609-9c1e-37b58c1e4bfb-plugins-config-volume\") pod \"40617548-d684-4609-9c1e-37b58c1e4bfb\" (UID: \"40617548-d684-4609-9c1e-37b58c1e4bfb\") " Oct 30 19:02:08.275511 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.275499 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40617548-d684-4609-9c1e-37b58c1e4bfb-kube-api-access-9p5xk\" (UniqueName: \"kubernetes.io/projected/40617548-d684-4609-9c1e-37b58c1e4bfb-kube-api-access-9p5xk\") pod \"40617548-d684-4609-9c1e-37b58c1e4bfb\" (UID: \"40617548-d684-4609-9c1e-37b58c1e4bfb\") " Oct 30 19:02:08.275723 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.275700 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40617548-d684-4609-9c1e-37b58c1e4bfb-plugins-config-volume" pod "40617548-d684-4609-9c1e-37b58c1e4bfb" (UID: "40617548-d684-4609-9c1e-37b58c1e4bfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:02:08.276106 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.276089 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40617548-d684-4609-9c1e-37b58c1e4bfb-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:08.276162 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.276112 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4rlq\" (UniqueName: \"kubernetes.io/projected/c6812606-a602-4a6d-b226-4be22c94d655-kube-api-access-b4rlq\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:08.277854 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.277835 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40617548-d684-4609-9c1e-37b58c1e4bfb-kube-api-access-9p5xk" pod "40617548-d684-4609-9c1e-37b58c1e4bfb" (UID: "40617548-d684-4609-9c1e-37b58c1e4bfb"). InnerVolumeSpecName "kube-api-access-9p5xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:02:08.378411 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.378312 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p5xk\" (UniqueName: \"kubernetes.io/projected/40617548-d684-4609-9c1e-37b58c1e4bfb-kube-api-access-9p5xk\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:08.470291 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.470250 4531 generic.go:356] "Generic (PLEG): container finished" podID="40617548-d684-4609-9c1e-37b58c1e4bfb" containerID="234635f6e6d8cc7e59acc6812c799418644a264a39159206dc5a07d8c2fd9879" exitCode=0 Oct 30 19:02:08.470492 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.470309 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t" Oct 30 19:02:08.470492 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.470346 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t" event={"ID":"40617548-d684-4609-9c1e-37b58c1e4bfb","Type":"ContainerDied","Data":"234635f6e6d8cc7e59acc6812c799418644a264a39159206dc5a07d8c2fd9879"} Oct 30 19:02:08.470492 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.470405 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t" event={"ID":"40617548-d684-4609-9c1e-37b58c1e4bfb","Type":"ContainerDied","Data":"fec6c203bfd4fd4f186f014b38fccd9fd3385ec1c176a84b983a511217e156cf"} Oct 30 19:02:08.470492 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.470435 4531 scope.go:122] "RemoveContainer" containerID="234635f6e6d8cc7e59acc6812c799418644a264a39159206dc5a07d8c2fd9879" Oct 30 19:02:08.471948 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.471922 4531 generic.go:356] "Generic (PLEG): container finished" podID="c6812606-a602-4a6d-b226-4be22c94d655" containerID="5a31c070b8016132f34144a38afb9ac313dcff0f74d3dc8dfa5ca56692c74ab7" exitCode=0 Oct 30 19:02:08.472063 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.471984 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn" Oct 30 19:02:08.472063 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.472013 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn" event={"ID":"c6812606-a602-4a6d-b226-4be22c94d655","Type":"ContainerDied","Data":"5a31c070b8016132f34144a38afb9ac313dcff0f74d3dc8dfa5ca56692c74ab7"} Oct 30 19:02:08.472179 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.472075 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn" event={"ID":"c6812606-a602-4a6d-b226-4be22c94d655","Type":"ContainerDied","Data":"7b7002d6affd95ed8131cfbd4ad2bbf61bcf1cb72dab5e4afaacfae98cf4cd32"} Oct 30 19:02:08.483726 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.483700 4531 scope.go:122] "RemoveContainer" containerID="234635f6e6d8cc7e59acc6812c799418644a264a39159206dc5a07d8c2fd9879" Oct 30 19:02:08.484055 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:02:08.484031 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"234635f6e6d8cc7e59acc6812c799418644a264a39159206dc5a07d8c2fd9879\": container with ID starting with 234635f6e6d8cc7e59acc6812c799418644a264a39159206dc5a07d8c2fd9879 not found: ID does not exist" containerID="234635f6e6d8cc7e59acc6812c799418644a264a39159206dc5a07d8c2fd9879" Oct 30 19:02:08.484187 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.484065 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"234635f6e6d8cc7e59acc6812c799418644a264a39159206dc5a07d8c2fd9879"} err="failed to get container status \"234635f6e6d8cc7e59acc6812c799418644a264a39159206dc5a07d8c2fd9879\": rpc error: code = NotFound desc = could not find container \"234635f6e6d8cc7e59acc6812c799418644a264a39159206dc5a07d8c2fd9879\": container with ID starting with 234635f6e6d8cc7e59acc6812c799418644a264a39159206dc5a07d8c2fd9879 not found: ID does not exist" Oct 30 19:02:08.484187 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.484084 4531 scope.go:122] "RemoveContainer" containerID="5a31c070b8016132f34144a38afb9ac313dcff0f74d3dc8dfa5ca56692c74ab7" Oct 30 19:02:08.494676 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.494652 4531 scope.go:122] "RemoveContainer" containerID="5a31c070b8016132f34144a38afb9ac313dcff0f74d3dc8dfa5ca56692c74ab7" Oct 30 19:02:08.495031 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:02:08.495011 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a31c070b8016132f34144a38afb9ac313dcff0f74d3dc8dfa5ca56692c74ab7\": container with ID starting with 5a31c070b8016132f34144a38afb9ac313dcff0f74d3dc8dfa5ca56692c74ab7 not found: ID does not exist" containerID="5a31c070b8016132f34144a38afb9ac313dcff0f74d3dc8dfa5ca56692c74ab7" Oct 30 19:02:08.495107 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.495041 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a31c070b8016132f34144a38afb9ac313dcff0f74d3dc8dfa5ca56692c74ab7"} err="failed to get container status \"5a31c070b8016132f34144a38afb9ac313dcff0f74d3dc8dfa5ca56692c74ab7\": rpc error: code = NotFound desc = could not find container \"5a31c070b8016132f34144a38afb9ac313dcff0f74d3dc8dfa5ca56692c74ab7\": container with ID starting with 5a31c070b8016132f34144a38afb9ac313dcff0f74d3dc8dfa5ca56692c74ab7 not found: ID does not exist" Oct 30 19:02:08.635712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.635623 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t"] Oct 30 19:02:08.642882 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.642845 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-66baf45c-6794b75d75-k9l8t"] Oct 30 19:02:08.753907 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.753871 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn"] Oct 30 19:02:08.761274 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:08.761239 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-66baf45c-59775bc55d-ws4pn"] Oct 30 19:02:09.516254 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.516221 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg"] Oct 30 19:02:09.516880 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.516860 4531 cpu_manager.go:414] "containerMap: removing container" podUID="40617548-d684-4609-9c1e-37b58c1e4bfb" containerName="payload-pre-processing" Oct 30 19:02:09.516938 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.516883 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40617548-d684-4609-9c1e-37b58c1e4bfb" containerName="payload-pre-processing" Oct 30 19:02:09.516938 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.516907 4531 cpu_manager.go:414] "containerMap: removing container" podUID="f07706fa-4548-44ac-8b1f-bcad8a391dfd" containerName="revoke-keys" Oct 30 19:02:09.516938 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.516914 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f07706fa-4548-44ac-8b1f-bcad8a391dfd" containerName="revoke-keys" Oct 30 19:02:09.516938 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.516929 4531 cpu_manager.go:414] "containerMap: removing container" podUID="8974a32e-1a13-435a-b4cc-1948881c3532" containerName="payload-pre-processing" Oct 30 19:02:09.516938 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.516934 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8974a32e-1a13-435a-b4cc-1948881c3532" containerName="payload-pre-processing" Oct 30 19:02:09.517117 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.516942 4531 cpu_manager.go:414] "containerMap: removing container" podUID="a58321dd-1bd5-45e5-89d1-1a9a60aaadf9" containerName="payload-processing" Oct 30 19:02:09.517117 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.516948 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a58321dd-1bd5-45e5-89d1-1a9a60aaadf9" containerName="payload-processing" Oct 30 19:02:09.517117 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.516969 4531 cpu_manager.go:414] "containerMap: removing container" podUID="c6812606-a602-4a6d-b226-4be22c94d655" containerName="payload-processing" Oct 30 19:02:09.517117 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.516976 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6812606-a602-4a6d-b226-4be22c94d655" containerName="payload-processing" Oct 30 19:02:09.517117 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.516990 4531 cpu_manager.go:414] "containerMap: removing container" podUID="a860f1c1-b883-44ba-bc53-e20c606ae368" containerName="payload-pre-processing" Oct 30 19:02:09.517117 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.516995 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a860f1c1-b883-44ba-bc53-e20c606ae368" containerName="payload-pre-processing" Oct 30 19:02:09.517117 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.517092 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f07706fa-4548-44ac-8b1f-bcad8a391dfd" containerName="revoke-keys" Oct 30 19:02:09.517117 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.517101 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8974a32e-1a13-435a-b4cc-1948881c3532" containerName="payload-pre-processing" Oct 30 19:02:09.517117 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.517106 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a58321dd-1bd5-45e5-89d1-1a9a60aaadf9" containerName="payload-processing" Oct 30 19:02:09.517117 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.517114 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6812606-a602-4a6d-b226-4be22c94d655" containerName="payload-processing" Oct 30 19:02:09.517117 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.517141 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a860f1c1-b883-44ba-bc53-e20c606ae368" containerName="payload-pre-processing" Oct 30 19:02:09.517534 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.517153 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="40617548-d684-4609-9c1e-37b58c1e4bfb" containerName="payload-pre-processing" Oct 30 19:02:09.521914 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.521891 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.595505 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.595465 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-workload-socket\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.595717 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.595517 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-credential-socket\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.595717 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.595662 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0005aee3-3a38-431e-b86c-d1c40bab8483-istiod-ca-cert\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.595717 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.595709 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl62d\" (UniqueName: \"kubernetes.io/projected/0005aee3-3a38-431e-b86c-d1c40bab8483-kube-api-access-sl62d\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.595872 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.595814 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-podinfo\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.595872 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.595867 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-workload-certs\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.596026 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.595939 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-data\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.596026 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.595967 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-envoy\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.596143 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.596056 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-token\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.697277 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.697235 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-credential-socket\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.697461 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.697333 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0005aee3-3a38-431e-b86c-d1c40bab8483-istiod-ca-cert\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.697461 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.697355 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl62d\" (UniqueName: \"kubernetes.io/projected/0005aee3-3a38-431e-b86c-d1c40bab8483-kube-api-access-sl62d\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.697461 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.697413 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-podinfo\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.697656 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.697471 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-workload-certs\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.697656 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.697513 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-data\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.697656 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.697552 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-envoy\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.697656 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.697631 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-token\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.697874 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.697681 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-workload-socket\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.697874 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.697716 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-credential-socket\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.697991 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.697977 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-data\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.698193 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.698122 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0005aee3-3a38-431e-b86c-d1c40bab8483-istiod-ca-cert\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.698296 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.698258 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-workload-certs\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.698347 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.698315 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-workload-socket\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.700461 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.700438 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-envoy\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.700547 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.700448 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-podinfo\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.723775 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.723740 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg"] Oct 30 19:02:09.758668 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.758626 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-token\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.758951 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.758915 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl62d\" (UniqueName: \"kubernetes.io/projected/0005aee3-3a38-431e-b86c-d1c40bab8483-kube-api-access-sl62d\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:09.845456 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:09.845350 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:10.099394 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:10.099313 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40617548-d684-4609-9c1e-37b58c1e4bfb" path="/var/lib/kubelet/pods/40617548-d684-4609-9c1e-37b58c1e4bfb/volumes" Oct 30 19:02:10.099769 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:10.099755 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6812606-a602-4a6d-b226-4be22c94d655" path="/var/lib/kubelet/pods/c6812606-a602-4a6d-b226-4be22c94d655/volumes" Oct 30 19:02:10.283059 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:10.283029 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg"] Oct 30 19:02:10.320292 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:02:10.320149 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0005aee3_3a38_431e_b86c_d1c40bab8483.slice/crio-30b8ac56c0e960fdac4578828c78d2903237c9c9cdbe58b5bee3d672afb0f56b WatchSource:0}: Error finding container 30b8ac56c0e960fdac4578828c78d2903237c9c9cdbe58b5bee3d672afb0f56b: Status 404 returned error can't find the container with id 30b8ac56c0e960fdac4578828c78d2903237c9c9cdbe58b5bee3d672afb0f56b Oct 30 19:02:10.322393 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:10.322358 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:02:10.322480 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:10.322428 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:02:10.322480 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:10.322459 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:02:10.494232 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:10.494193 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" event={"ID":"0005aee3-3a38-431e-b86c-d1c40bab8483","Type":"ContainerStarted","Data":"6740d54ef5d500dad169fdbb4059b728df2209f27c3952d009654f7a9fdac402"} Oct 30 19:02:10.494416 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:10.494272 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" event={"ID":"0005aee3-3a38-431e-b86c-d1c40bab8483","Type":"ContainerStarted","Data":"30b8ac56c0e960fdac4578828c78d2903237c9c9cdbe58b5bee3d672afb0f56b"} Oct 30 19:02:10.617148 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:10.617073 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" podStartSLOduration=1.6170560219999999 podStartE2EDuration="1.617056022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:10.613223454 +0000 UTC m=+1690.640328415" watchObservedRunningTime="2026-10-30 19:02:10.617056022 +0000 UTC m=+1690.644160981" Oct 30 19:02:10.845573 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:10.845534 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:11.847361 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:11.847324 4531 patch_prober.go:29] interesting pod/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 19:02:11.847810 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:11.847386 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" podUID="0005aee3-3a38-431e-b86c-d1c40bab8483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 19:02:12.846457 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:12.846421 4531 patch_prober.go:29] interesting pod/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 19:02:12.846649 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:12.846480 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" podUID="0005aee3-3a38-431e-b86c-d1c40bab8483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 19:02:13.846792 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:13.846748 4531 patch_prober.go:29] interesting pod/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 19:02:13.847336 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:13.846832 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" podUID="0005aee3-3a38-431e-b86c-d1c40bab8483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 19:02:14.846684 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:14.846651 4531 patch_prober.go:29] interesting pod/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 19:02:14.846875 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:14.846719 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" podUID="0005aee3-3a38-431e-b86c-d1c40bab8483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 19:02:15.845843 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:15.845809 4531 patch_prober.go:29] interesting pod/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 19:02:15.846027 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:15.845876 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" podUID="0005aee3-3a38-431e-b86c-d1c40bab8483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 19:02:16.850572 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:16.850539 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:16.850988 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:16.850974 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:16.851810 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:16.851789 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:02:24.623429 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.623397 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg"] Oct 30 19:02:24.627491 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.627463 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.725179 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.725091 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg"] Oct 30 19:02:24.740765 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.740053 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-data\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.740765 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.740119 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-token\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.740765 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.740182 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed7421d-881e-4a42-a128-e4852abe4cfd-istiod-ca-cert\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.740765 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.740210 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-podinfo\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.740765 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.740243 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-workload-certs\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.740765 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.740272 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gngv4\" (UniqueName: \"kubernetes.io/projected/2ed7421d-881e-4a42-a128-e4852abe4cfd-kube-api-access-gngv4\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.740765 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.740314 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-credential-socket\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.740765 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.740335 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-envoy\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.740765 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.740512 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-workload-socket\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.843197 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.843058 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-data\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.843197 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.843158 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-token\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.843197 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.843207 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed7421d-881e-4a42-a128-e4852abe4cfd-istiod-ca-cert\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.843552 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.843238 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-podinfo\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.843552 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.843280 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-workload-certs\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.843552 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.843314 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gngv4\" (UniqueName: \"kubernetes.io/projected/2ed7421d-881e-4a42-a128-e4852abe4cfd-kube-api-access-gngv4\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.843552 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.843360 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-credential-socket\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.843552 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.843386 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-envoy\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.843810 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.843579 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-workload-socket\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.844246 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.844120 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-workload-socket\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.844590 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.844453 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-data\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.848549 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.847722 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed7421d-881e-4a42-a128-e4852abe4cfd-istiod-ca-cert\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.849779 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.849748 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-credential-socket\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.853265 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.850221 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-workload-certs\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.865885 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.865844 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-podinfo\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.892926 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.892847 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-envoy\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.910441 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.910398 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-token\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.913254 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.913223 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gngv4\" (UniqueName: \"kubernetes.io/projected/2ed7421d-881e-4a42-a128-e4852abe4cfd-kube-api-access-gngv4\") pod \"e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:24.946036 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:24.941957 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:25.395505 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:25.395465 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg"] Oct 30 19:02:25.396501 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:02:25.396468 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed7421d_881e_4a42_a128_e4852abe4cfd.slice/crio-520b17a557812536591d9dfca2c0f664b6a5671a47396a07e9edcf4032fff012 WatchSource:0}: Error finding container 520b17a557812536591d9dfca2c0f664b6a5671a47396a07e9edcf4032fff012: Status 404 returned error can't find the container with id 520b17a557812536591d9dfca2c0f664b6a5671a47396a07e9edcf4032fff012 Oct 30 19:02:25.403556 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:25.402653 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:02:25.403556 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:25.402754 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:02:25.403556 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:25.402800 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:02:25.636445 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:25.636409 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" event={"ID":"2ed7421d-881e-4a42-a128-e4852abe4cfd","Type":"ContainerStarted","Data":"43a351f28a72b65aa3439ddbb0d834d61a721caf1011386af9ace1ed9f907d6e"} Oct 30 19:02:25.636912 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:25.636476 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" event={"ID":"2ed7421d-881e-4a42-a128-e4852abe4cfd","Type":"ContainerStarted","Data":"520b17a557812536591d9dfca2c0f664b6a5671a47396a07e9edcf4032fff012"} Oct 30 19:02:25.761408 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:25.761334 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" podStartSLOduration=1.761315921 podStartE2EDuration="1.761315921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:25.756706112 +0000 UTC m=+1705.783811073" watchObservedRunningTime="2026-10-30 19:02:25.761315921 +0000 UTC m=+1705.788420879" Oct 30 19:02:25.912256 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:25.912202 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8"] Oct 30 19:02:25.915974 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:25.915940 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66baf45c-kdbc8"] Oct 30 19:02:25.950231 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:25.943229 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:26.182210 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:26.182038 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f07706fa-4548-44ac-8b1f-bcad8a391dfd" path="/var/lib/kubelet/pods/f07706fa-4548-44ac-8b1f-bcad8a391dfd/volumes" Oct 30 19:02:26.500157 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:02:26.497581 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:02:26.945089 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:26.945001 4531 patch_prober.go:29] interesting pod/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 19:02:26.945089 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:26.945059 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" podUID="2ed7421d-881e-4a42-a128-e4852abe4cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 19:02:27.181892 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:27.181847 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-85ab6a-mql96"] Oct 30 19:02:27.241923 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:27.241799 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-85ab6a-mql96"] Oct 30 19:02:27.943094 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:27.943053 4531 patch_prober.go:29] interesting pod/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 19:02:27.943338 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:27.943166 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" podUID="2ed7421d-881e-4a42-a128-e4852abe4cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 19:02:28.099607 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:28.099572 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb908afb-d997-4479-997f-be8958658b69" path="/var/lib/kubelet/pods/eb908afb-d997-4479-997f-be8958658b69/volumes" Oct 30 19:02:28.943953 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:28.943910 4531 patch_prober.go:29] interesting pod/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 19:02:28.944179 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:28.943979 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" podUID="2ed7421d-881e-4a42-a128-e4852abe4cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 19:02:29.943861 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:29.943819 4531 patch_prober.go:29] interesting pod/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 19:02:29.944506 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:29.943902 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" podUID="2ed7421d-881e-4a42-a128-e4852abe4cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 19:02:30.943710 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:30.943671 4531 patch_prober.go:29] interesting pod/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 19:02:30.943920 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:30.943748 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" podUID="2ed7421d-881e-4a42-a128-e4852abe4cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 19:02:31.344270 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:31.344226 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx"] Oct 30 19:02:31.967223 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:31.967187 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:31.967676 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:31.967624 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:31.968467 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:31.968451 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:02:32.072603 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:32.072565 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3e044f1-tctmr"] Oct 30 19:02:32.075741 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:32.075717 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3e044f1-tctmr"] Oct 30 19:02:32.101930 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:32.101898 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e26a1ca-1ae0-4609-bf0f-a366f9462bb4" path="/var/lib/kubelet/pods/5e26a1ca-1ae0-4609-bf0f-a366f9462bb4/volumes" Oct 30 19:02:35.226199 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:35.226158 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-132af1-zsd7z"] Oct 30 19:02:35.231517 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:35.231498 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-132af1-zsd7z" Oct 30 19:02:35.296453 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:35.296420 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-132af1-zsd7z"] Oct 30 19:02:35.341523 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:35.341483 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:02:35.341523 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:35.341520 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:02:35.341523 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:35.341497 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:02:35.341782 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:35.341492 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:02:35.362538 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:35.362498 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2ed0c4-859a-41eb-9391-8720e3b58365-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-132af1-zsd7z\" (UID: \"2b2ed0c4-859a-41eb-9391-8720e3b58365\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-132af1-zsd7z" Oct 30 19:02:35.362713 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:35.362556 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2ed0c4-859a-41eb-9391-8720e3b58365-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-132af1-zsd7z\" (UID: \"2b2ed0c4-859a-41eb-9391-8720e3b58365\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-132af1-zsd7z" Oct 30 19:02:35.362826 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:35.362792 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dndl\" (UniqueName: \"kubernetes.io/projected/2b2ed0c4-859a-41eb-9391-8720e3b58365-kube-api-access-7dndl\") pod \"maas-api-revoke-keys-e2e-api-b-132af1-zsd7z\" (UID: \"2b2ed0c4-859a-41eb-9391-8720e3b58365\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-132af1-zsd7z" Oct 30 19:02:35.464277 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:35.464240 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dndl\" (UniqueName: \"kubernetes.io/projected/2b2ed0c4-859a-41eb-9391-8720e3b58365-kube-api-access-7dndl\") pod \"maas-api-revoke-keys-e2e-api-b-132af1-zsd7z\" (UID: \"2b2ed0c4-859a-41eb-9391-8720e3b58365\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-132af1-zsd7z" Oct 30 19:02:35.464473 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:35.464425 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2ed0c4-859a-41eb-9391-8720e3b58365-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-132af1-zsd7z\" (UID: \"2b2ed0c4-859a-41eb-9391-8720e3b58365\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-132af1-zsd7z" Oct 30 19:02:35.464535 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:35.464477 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2ed0c4-859a-41eb-9391-8720e3b58365-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-132af1-zsd7z\" (UID: \"2b2ed0c4-859a-41eb-9391-8720e3b58365\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-132af1-zsd7z" Oct 30 19:02:35.465146 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:35.465108 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2ed0c4-859a-41eb-9391-8720e3b58365-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-132af1-zsd7z\" (UID: \"2b2ed0c4-859a-41eb-9391-8720e3b58365\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-132af1-zsd7z" Oct 30 19:02:35.467035 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:35.467010 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2ed0c4-859a-41eb-9391-8720e3b58365-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-132af1-zsd7z\" (UID: \"2b2ed0c4-859a-41eb-9391-8720e3b58365\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-132af1-zsd7z" Oct 30 19:02:35.525169 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:35.525114 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dndl\" (UniqueName: \"kubernetes.io/projected/2b2ed0c4-859a-41eb-9391-8720e3b58365-kube-api-access-7dndl\") pod \"maas-api-revoke-keys-e2e-api-b-132af1-zsd7z\" (UID: \"2b2ed0c4-859a-41eb-9391-8720e3b58365\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-132af1-zsd7z" Oct 30 19:02:35.546448 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:35.546425 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-132af1-zsd7z" Oct 30 19:02:35.927644 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:35.927617 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-132af1-zsd7z"] Oct 30 19:02:35.928883 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:02:35.928859 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2ed0c4_859a_41eb_9391_8720e3b58365.slice/crio-aeb4939f03013fe8ba4fcae08b052497800794873aa69fc97157e78f52a9ff08 WatchSource:0}: Error finding container aeb4939f03013fe8ba4fcae08b052497800794873aa69fc97157e78f52a9ff08: Status 404 returned error can't find the container with id aeb4939f03013fe8ba4fcae08b052497800794873aa69fc97157e78f52a9ff08 Oct 30 19:02:36.793113 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:36.793085 4531 generic.go:356] "Generic (PLEG): container finished" podID="2b2ed0c4-859a-41eb-9391-8720e3b58365" containerID="13c2c9bc41b55d970049a251471eb060b7c86f2ec3247151b50ef0d5860ea824" exitCode=0 Oct 30 19:02:36.793591 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:36.793211 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-132af1-zsd7z" event={"ID":"2b2ed0c4-859a-41eb-9391-8720e3b58365","Type":"ContainerDied","Data":"13c2c9bc41b55d970049a251471eb060b7c86f2ec3247151b50ef0d5860ea824"} Oct 30 19:02:36.793591 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:36.793249 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-132af1-zsd7z" event={"ID":"2b2ed0c4-859a-41eb-9391-8720e3b58365","Type":"ContainerStarted","Data":"aeb4939f03013fe8ba4fcae08b052497800794873aa69fc97157e78f52a9ff08"} Oct 30 19:02:37.967609 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:37.967578 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-132af1-zsd7z" Oct 30 19:02:38.104889 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:38.104794 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b2ed0c4-859a-41eb-9391-8720e3b58365-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2ed0c4-859a-41eb-9391-8720e3b58365-maas-api-xks-ca\") pod \"2b2ed0c4-859a-41eb-9391-8720e3b58365\" (UID: \"2b2ed0c4-859a-41eb-9391-8720e3b58365\") " Oct 30 19:02:38.105231 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:38.105203 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b2ed0c4-859a-41eb-9391-8720e3b58365-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2ed0c4-859a-41eb-9391-8720e3b58365-maas-api-openshift-ca\") pod \"2b2ed0c4-859a-41eb-9391-8720e3b58365\" (UID: \"2b2ed0c4-859a-41eb-9391-8720e3b58365\") " Oct 30 19:02:38.105394 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:38.105247 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b2ed0c4-859a-41eb-9391-8720e3b58365-kube-api-access-7dndl\" (UniqueName: \"kubernetes.io/projected/2b2ed0c4-859a-41eb-9391-8720e3b58365-kube-api-access-7dndl\") pod \"2b2ed0c4-859a-41eb-9391-8720e3b58365\" (UID: \"2b2ed0c4-859a-41eb-9391-8720e3b58365\") " Oct 30 19:02:38.105622 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:38.105595 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b2ed0c4-859a-41eb-9391-8720e3b58365-maas-api-openshift-ca" pod "2b2ed0c4-859a-41eb-9391-8720e3b58365" (UID: "2b2ed0c4-859a-41eb-9391-8720e3b58365"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:02:38.106095 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:38.106070 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2ed0c4-859a-41eb-9391-8720e3b58365-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:38.108310 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:38.108276 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2ed0c4-859a-41eb-9391-8720e3b58365-kube-api-access-7dndl" pod "2b2ed0c4-859a-41eb-9391-8720e3b58365" (UID: "2b2ed0c4-859a-41eb-9391-8720e3b58365"). InnerVolumeSpecName "kube-api-access-7dndl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:02:38.108454 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:38.108428 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b2ed0c4-859a-41eb-9391-8720e3b58365-maas-api-xks-ca" pod "2b2ed0c4-859a-41eb-9391-8720e3b58365" (UID: "2b2ed0c4-859a-41eb-9391-8720e3b58365"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:02:38.207451 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:38.207401 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2ed0c4-859a-41eb-9391-8720e3b58365-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:38.207451 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:38.207435 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dndl\" (UniqueName: \"kubernetes.io/projected/2b2ed0c4-859a-41eb-9391-8720e3b58365-kube-api-access-7dndl\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:38.813079 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:38.813045 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-132af1-zsd7z" event={"ID":"2b2ed0c4-859a-41eb-9391-8720e3b58365","Type":"ContainerDied","Data":"aeb4939f03013fe8ba4fcae08b052497800794873aa69fc97157e78f52a9ff08"} Oct 30 19:02:38.813282 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:38.813101 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aeb4939f03013fe8ba4fcae08b052497800794873aa69fc97157e78f52a9ff08" Oct 30 19:02:38.813282 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:38.813118 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-132af1-zsd7z" Oct 30 19:02:43.786963 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:43.786925 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88"] Oct 30 19:02:43.787466 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:43.787180 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" podUID="88fc33e1-96f8-4682-aedf-ef669f85138c" containerName="istio-proxy" containerID="cri-o://940cbb9dabd7da95639c197d9454b7036ce152514e07bfd83f9241f66485b38b" gracePeriod=30 Oct 30 19:02:46.366744 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.366705 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" podUID="dab47e85-278b-4d4f-94b2-8f46ec744d69" containerName="main" containerID="cri-o://8328d8383b5c5d699465f3924a5e6c2a8c764a0347d102ded9a2fc8e5d069ad1" gracePeriod=45 Oct 30 19:02:46.759479 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.759446 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" Oct 30 19:02:46.863175 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.863092 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-tmp-dir\") pod \"dab47e85-278b-4d4f-94b2-8f46ec744d69\" (UID: \"dab47e85-278b-4d4f-94b2-8f46ec744d69\") " Oct 30 19:02:46.863175 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.863170 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-dshm\") pod \"dab47e85-278b-4d4f-94b2-8f46ec744d69\" (UID: \"dab47e85-278b-4d4f-94b2-8f46ec744d69\") " Oct 30 19:02:46.863445 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.863274 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-model-cache\") pod \"dab47e85-278b-4d4f-94b2-8f46ec744d69\" (UID: \"dab47e85-278b-4d4f-94b2-8f46ec744d69\") " Oct 30 19:02:46.863445 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.863353 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dab47e85-278b-4d4f-94b2-8f46ec744d69-tls-certs\" (UniqueName: \"kubernetes.io/secret/dab47e85-278b-4d4f-94b2-8f46ec744d69-tls-certs\") pod \"dab47e85-278b-4d4f-94b2-8f46ec744d69\" (UID: \"dab47e85-278b-4d4f-94b2-8f46ec744d69\") " Oct 30 19:02:46.863600 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.863445 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-tmp-dir" pod "dab47e85-278b-4d4f-94b2-8f46ec744d69" (UID: "dab47e85-278b-4d4f-94b2-8f46ec744d69"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:46.863600 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.863459 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-home\" (UniqueName: \"kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-home\") pod \"dab47e85-278b-4d4f-94b2-8f46ec744d69\" (UID: \"dab47e85-278b-4d4f-94b2-8f46ec744d69\") " Oct 30 19:02:46.863600 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.863535 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dab47e85-278b-4d4f-94b2-8f46ec744d69-kube-api-access-wmz97\" (UniqueName: \"kubernetes.io/projected/dab47e85-278b-4d4f-94b2-8f46ec744d69-kube-api-access-wmz97\") pod \"dab47e85-278b-4d4f-94b2-8f46ec744d69\" (UID: \"dab47e85-278b-4d4f-94b2-8f46ec744d69\") " Oct 30 19:02:46.863766 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.863624 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-model-cache" pod "dab47e85-278b-4d4f-94b2-8f46ec744d69" (UID: "dab47e85-278b-4d4f-94b2-8f46ec744d69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:46.863826 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.863773 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-home" pod "dab47e85-278b-4d4f-94b2-8f46ec744d69" (UID: "dab47e85-278b-4d4f-94b2-8f46ec744d69"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:46.864531 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.864501 4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-model-cache\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:46.864531 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.864533 4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-home\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:46.864763 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.864549 4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-tmp-dir\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:46.866253 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.866217 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dab47e85-278b-4d4f-94b2-8f46ec744d69-tls-certs" pod "dab47e85-278b-4d4f-94b2-8f46ec744d69" (UID: "dab47e85-278b-4d4f-94b2-8f46ec744d69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:02:46.866374 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.866265 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-dshm" pod "dab47e85-278b-4d4f-94b2-8f46ec744d69" (UID: "dab47e85-278b-4d4f-94b2-8f46ec744d69"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:46.866374 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.866319 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab47e85-278b-4d4f-94b2-8f46ec744d69-kube-api-access-wmz97" pod "dab47e85-278b-4d4f-94b2-8f46ec744d69" (UID: "dab47e85-278b-4d4f-94b2-8f46ec744d69"). InnerVolumeSpecName "kube-api-access-wmz97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:02:46.873337 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.869445 4531 generic.go:356] "Generic (PLEG): container finished" podID="dab47e85-278b-4d4f-94b2-8f46ec744d69" containerID="8328d8383b5c5d699465f3924a5e6c2a8c764a0347d102ded9a2fc8e5d069ad1" exitCode=0 Oct 30 19:02:46.873337 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.869821 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" event={"ID":"dab47e85-278b-4d4f-94b2-8f46ec744d69","Type":"ContainerDied","Data":"8328d8383b5c5d699465f3924a5e6c2a8c764a0347d102ded9a2fc8e5d069ad1"} Oct 30 19:02:46.873337 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.869881 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" event={"ID":"dab47e85-278b-4d4f-94b2-8f46ec744d69","Type":"ContainerDied","Data":"94a0f09b441caacea8e69b4962b0eaafe399033af197127ede4bb94510217dd1"} Oct 30 19:02:46.873337 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.869906 4531 scope.go:122] "RemoveContainer" containerID="8328d8383b5c5d699465f3924a5e6c2a8c764a0347d102ded9a2fc8e5d069ad1" Oct 30 19:02:46.873337 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.870058 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx" Oct 30 19:02:46.897692 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.897662 4531 scope.go:122] "RemoveContainer" containerID="8328d8383b5c5d699465f3924a5e6c2a8c764a0347d102ded9a2fc8e5d069ad1" Oct 30 19:02:46.898110 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:02:46.898087 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8328d8383b5c5d699465f3924a5e6c2a8c764a0347d102ded9a2fc8e5d069ad1\": container with ID starting with 8328d8383b5c5d699465f3924a5e6c2a8c764a0347d102ded9a2fc8e5d069ad1 not found: ID does not exist" containerID="8328d8383b5c5d699465f3924a5e6c2a8c764a0347d102ded9a2fc8e5d069ad1" Oct 30 19:02:46.898328 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.898124 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8328d8383b5c5d699465f3924a5e6c2a8c764a0347d102ded9a2fc8e5d069ad1"} err="failed to get container status \"8328d8383b5c5d699465f3924a5e6c2a8c764a0347d102ded9a2fc8e5d069ad1\": rpc error: code = NotFound desc = could not find container \"8328d8383b5c5d699465f3924a5e6c2a8c764a0347d102ded9a2fc8e5d069ad1\": container with ID starting with 8328d8383b5c5d699465f3924a5e6c2a8c764a0347d102ded9a2fc8e5d069ad1 not found: ID does not exist" Oct 30 19:02:46.932623 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.932582 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47"] Oct 30 19:02:46.965513 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.965471 4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dab47e85-278b-4d4f-94b2-8f46ec744d69-tls-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:46.965513 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.965509 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmz97\" (UniqueName: \"kubernetes.io/projected/dab47e85-278b-4d4f-94b2-8f46ec744d69-kube-api-access-wmz97\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:46.965513 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:46.965520 4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dab47e85-278b-4d4f-94b2-8f46ec744d69-dshm\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:47.044323 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:47.044284 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx"] Oct 30 19:02:47.052550 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:47.052513 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-66baf45c/e2e-lifecycle-model-66baf45c-kserve-75c7cb457-7clxx"] Oct 30 19:02:48.099751 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:48.099715 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dab47e85-278b-4d4f-94b2-8f46ec744d69" path="/var/lib/kubelet/pods/dab47e85-278b-4d4f-94b2-8f46ec744d69/volumes" Oct 30 19:02:49.066544 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.066476 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 19:02:49.105373 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.105331 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88fc33e1-96f8-4682-aedf-ef669f85138c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88fc33e1-96f8-4682-aedf-ef669f85138c-istiod-ca-cert\") pod \"88fc33e1-96f8-4682-aedf-ef669f85138c\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " Oct 30 19:02:49.105859 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.105440 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88fc33e1-96f8-4682-aedf-ef669f85138c-kube-api-access-9ft4x\" (UniqueName: \"kubernetes.io/projected/88fc33e1-96f8-4682-aedf-ef669f85138c-kube-api-access-9ft4x\") pod \"88fc33e1-96f8-4682-aedf-ef669f85138c\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " Oct 30 19:02:49.105859 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.105476 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-workload-socket\") pod \"88fc33e1-96f8-4682-aedf-ef669f85138c\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " Oct 30 19:02:49.105859 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.105497 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-token\" (UniqueName: \"kubernetes.io/projected/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-token\") pod \"88fc33e1-96f8-4682-aedf-ef669f85138c\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " Oct 30 19:02:49.105859 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.105523 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-workload-certs\") pod \"88fc33e1-96f8-4682-aedf-ef669f85138c\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " Oct 30 19:02:49.105859 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.105578 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-data\") pod \"88fc33e1-96f8-4682-aedf-ef669f85138c\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " Oct 30 19:02:49.105859 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.105643 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-podinfo\") pod \"88fc33e1-96f8-4682-aedf-ef669f85138c\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " Oct 30 19:02:49.105859 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.105791 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88fc33e1-96f8-4682-aedf-ef669f85138c-istiod-ca-cert" pod "88fc33e1-96f8-4682-aedf-ef669f85138c" (UID: "88fc33e1-96f8-4682-aedf-ef669f85138c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:02:49.106445 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.105876 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-data" pod "88fc33e1-96f8-4682-aedf-ef669f85138c" (UID: "88fc33e1-96f8-4682-aedf-ef669f85138c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:49.106445 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.105785 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-workload-socket" pod "88fc33e1-96f8-4682-aedf-ef669f85138c" (UID: "88fc33e1-96f8-4682-aedf-ef669f85138c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:49.106445 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.105887 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-workload-certs" pod "88fc33e1-96f8-4682-aedf-ef669f85138c" (UID: "88fc33e1-96f8-4682-aedf-ef669f85138c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:49.106445 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.106023 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-credential-socket\") pod \"88fc33e1-96f8-4682-aedf-ef669f85138c\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " Oct 30 19:02:49.106445 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.106113 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-envoy\") pod \"88fc33e1-96f8-4682-aedf-ef669f85138c\" (UID: \"88fc33e1-96f8-4682-aedf-ef669f85138c\") " Oct 30 19:02:49.106445 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.106325 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-credential-socket" pod "88fc33e1-96f8-4682-aedf-ef669f85138c" (UID: "88fc33e1-96f8-4682-aedf-ef669f85138c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:49.106770 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.106676 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:49.106770 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.106694 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:49.106770 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.106703 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:49.106770 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.106713 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:49.106770 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.106721 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88fc33e1-96f8-4682-aedf-ef669f85138c-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:49.108075 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.108040 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88fc33e1-96f8-4682-aedf-ef669f85138c-kube-api-access-9ft4x" pod "88fc33e1-96f8-4682-aedf-ef669f85138c" (UID: "88fc33e1-96f8-4682-aedf-ef669f85138c"). InnerVolumeSpecName "kube-api-access-9ft4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:02:49.108075 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.108065 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-podinfo" pod "88fc33e1-96f8-4682-aedf-ef669f85138c" (UID: "88fc33e1-96f8-4682-aedf-ef669f85138c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:02:49.108440 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.108418 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-token" pod "88fc33e1-96f8-4682-aedf-ef669f85138c" (UID: "88fc33e1-96f8-4682-aedf-ef669f85138c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:02:49.108526 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.108512 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-envoy" pod "88fc33e1-96f8-4682-aedf-ef669f85138c" (UID: "88fc33e1-96f8-4682-aedf-ef669f85138c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:49.208238 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.208199 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:49.208238 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.208239 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ft4x\" (UniqueName: \"kubernetes.io/projected/88fc33e1-96f8-4682-aedf-ef669f85138c-kube-api-access-9ft4x\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:49.208448 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.208256 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:49.208448 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.208270 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88fc33e1-96f8-4682-aedf-ef669f85138c-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:49.895921 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.895883 4531 generic.go:356] "Generic (PLEG): container finished" podID="88fc33e1-96f8-4682-aedf-ef669f85138c" containerID="940cbb9dabd7da95639c197d9454b7036ce152514e07bfd83f9241f66485b38b" exitCode=0 Oct 30 19:02:49.896174 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.895936 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" Oct 30 19:02:49.896174 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.895970 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" event={"ID":"88fc33e1-96f8-4682-aedf-ef669f85138c","Type":"ContainerDied","Data":"940cbb9dabd7da95639c197d9454b7036ce152514e07bfd83f9241f66485b38b"} Oct 30 19:02:49.896174 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.896033 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" event={"ID":"88fc33e1-96f8-4682-aedf-ef669f85138c","Type":"ContainerDied","Data":"812c5e327a567bb9835d87334d24e13b2255390a91ce60d90873e213447bf29b"} Oct 30 19:02:49.896174 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.896064 4531 scope.go:122] "RemoveContainer" containerID="940cbb9dabd7da95639c197d9454b7036ce152514e07bfd83f9241f66485b38b" Oct 30 19:02:49.909411 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.909190 4531 scope.go:122] "RemoveContainer" containerID="940cbb9dabd7da95639c197d9454b7036ce152514e07bfd83f9241f66485b38b" Oct 30 19:02:49.909617 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:02:49.909588 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"940cbb9dabd7da95639c197d9454b7036ce152514e07bfd83f9241f66485b38b\": container with ID starting with 940cbb9dabd7da95639c197d9454b7036ce152514e07bfd83f9241f66485b38b not found: ID does not exist" containerID="940cbb9dabd7da95639c197d9454b7036ce152514e07bfd83f9241f66485b38b" Oct 30 19:02:49.909683 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:49.909630 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940cbb9dabd7da95639c197d9454b7036ce152514e07bfd83f9241f66485b38b"} err="failed to get container status \"940cbb9dabd7da95639c197d9454b7036ce152514e07bfd83f9241f66485b38b\": rpc error: code = NotFound desc = could not find container \"940cbb9dabd7da95639c197d9454b7036ce152514e07bfd83f9241f66485b38b\": container with ID starting with 940cbb9dabd7da95639c197d9454b7036ce152514e07bfd83f9241f66485b38b not found: ID does not exist" Oct 30 19:02:50.001760 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:50.001706 4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:02:50.001956 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:50.001782 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88" podUID="88fc33e1-96f8-4682-aedf-ef669f85138c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:02:50.113541 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:50.113502 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88"] Oct 30 19:02:50.119675 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:50.119640 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-6bm88"] Oct 30 19:02:51.934064 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:51.933984 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47" podUID="6f8455f6-97f8-47c7-b862-34d0a948329b" containerName="payload-pre-processing" containerID="cri-o://1d4b4dff91768fb5afc7b4176dcdf9465464203abc9df1289616113aba93eb0b" gracePeriod=25 Oct 30 19:02:52.105790 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:52.105752 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88fc33e1-96f8-4682-aedf-ef669f85138c" path="/var/lib/kubelet/pods/88fc33e1-96f8-4682-aedf-ef669f85138c/volumes" Oct 30 19:02:52.253226 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:52.253192 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47" Oct 30 19:02:52.353256 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:52.353225 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f8455f6-97f8-47c7-b862-34d0a948329b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f8455f6-97f8-47c7-b862-34d0a948329b-plugins-config-volume\") pod \"6f8455f6-97f8-47c7-b862-34d0a948329b\" (UID: \"6f8455f6-97f8-47c7-b862-34d0a948329b\") " Oct 30 19:02:52.353486 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:52.353309 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f8455f6-97f8-47c7-b862-34d0a948329b-kube-api-access-j6qst\" (UniqueName: \"kubernetes.io/projected/6f8455f6-97f8-47c7-b862-34d0a948329b-kube-api-access-j6qst\") pod \"6f8455f6-97f8-47c7-b862-34d0a948329b\" (UID: \"6f8455f6-97f8-47c7-b862-34d0a948329b\") " Oct 30 19:02:52.353714 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:52.353686 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f8455f6-97f8-47c7-b862-34d0a948329b-plugins-config-volume" pod "6f8455f6-97f8-47c7-b862-34d0a948329b" (UID: "6f8455f6-97f8-47c7-b862-34d0a948329b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:02:52.353894 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:52.353875 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f8455f6-97f8-47c7-b862-34d0a948329b-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:52.355791 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:52.355761 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f8455f6-97f8-47c7-b862-34d0a948329b-kube-api-access-j6qst" pod "6f8455f6-97f8-47c7-b862-34d0a948329b" (UID: "6f8455f6-97f8-47c7-b862-34d0a948329b"). InnerVolumeSpecName "kube-api-access-j6qst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:02:52.455809 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:52.455728 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6qst\" (UniqueName: \"kubernetes.io/projected/6f8455f6-97f8-47c7-b862-34d0a948329b-kube-api-access-j6qst\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:02:52.920643 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:52.920608 4531 generic.go:356] "Generic (PLEG): container finished" podID="6f8455f6-97f8-47c7-b862-34d0a948329b" containerID="1d4b4dff91768fb5afc7b4176dcdf9465464203abc9df1289616113aba93eb0b" exitCode=0 Oct 30 19:02:52.920841 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:52.920664 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47" Oct 30 19:02:52.920841 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:52.920681 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47" event={"ID":"6f8455f6-97f8-47c7-b862-34d0a948329b","Type":"ContainerDied","Data":"1d4b4dff91768fb5afc7b4176dcdf9465464203abc9df1289616113aba93eb0b"} Oct 30 19:02:52.920841 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:52.920739 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47" event={"ID":"6f8455f6-97f8-47c7-b862-34d0a948329b","Type":"ContainerDied","Data":"4493d476598e2171f65cccec57b08632895c459064b8e8444465baab58553f63"} Oct 30 19:02:52.920841 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:52.920762 4531 scope.go:122] "RemoveContainer" containerID="1d4b4dff91768fb5afc7b4176dcdf9465464203abc9df1289616113aba93eb0b" Oct 30 19:02:52.931044 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:52.931025 4531 scope.go:122] "RemoveContainer" containerID="1d4b4dff91768fb5afc7b4176dcdf9465464203abc9df1289616113aba93eb0b" Oct 30 19:02:52.931353 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:02:52.931333 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d4b4dff91768fb5afc7b4176dcdf9465464203abc9df1289616113aba93eb0b\": container with ID starting with 1d4b4dff91768fb5afc7b4176dcdf9465464203abc9df1289616113aba93eb0b not found: ID does not exist" containerID="1d4b4dff91768fb5afc7b4176dcdf9465464203abc9df1289616113aba93eb0b" Oct 30 19:02:52.931424 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:52.931360 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d4b4dff91768fb5afc7b4176dcdf9465464203abc9df1289616113aba93eb0b"} err="failed to get container status \"1d4b4dff91768fb5afc7b4176dcdf9465464203abc9df1289616113aba93eb0b\": rpc error: code = NotFound desc = could not find container \"1d4b4dff91768fb5afc7b4176dcdf9465464203abc9df1289616113aba93eb0b\": container with ID starting with 1d4b4dff91768fb5afc7b4176dcdf9465464203abc9df1289616113aba93eb0b not found: ID does not exist" Oct 30 19:02:53.085555 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:53.085524 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47"] Oct 30 19:02:53.134703 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:53.134664 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-132af1-85b5cc7b99-mkx47"] Oct 30 19:02:54.100580 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:02:54.100539 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f8455f6-97f8-47c7-b862-34d0a948329b" path="/var/lib/kubelet/pods/6f8455f6-97f8-47c7-b862-34d0a948329b/volumes" Oct 30 19:02:56.398804 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:02:56.398765 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:03:01.233760 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:01.233720 4531 scope.go:122] "RemoveContainer" containerID="3134c6f1984b1ffe0bbf01174bf1bf9d8d923cf76d67b78e33803f29d9c5b4fd" Oct 30 19:03:01.244286 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:01.244258 4531 scope.go:122] "RemoveContainer" containerID="fdf761f8a30c040544d6ad66342286e8eadbb85a8cf815981f0a6a84e63c3486" Oct 30 19:03:01.254961 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:01.254934 4531 scope.go:122] "RemoveContainer" containerID="468794aa7e638429af14aee6c342e56e2de19d8cdcf02ea675e97b6a5e06a1a6" Oct 30 19:03:13.119212 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:13.118825 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-470093-68ccl"] Oct 30 19:03:13.187160 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:13.167855 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-470093-68ccl"] Oct 30 19:03:14.099308 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:14.099276 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c1c263c-35c6-4224-a3f6-dc944720321b" path="/var/lib/kubelet/pods/1c1c263c-35c6-4224-a3f6-dc944720321b/volumes" Oct 30 19:03:14.448293 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:14.448211 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s"] Oct 30 19:03:14.497549 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:14.497506 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-83d5bd-k4m2s"] Oct 30 19:03:16.099529 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:16.099495 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6b35e72-807a-48b1-a764-9698ffe71dc2" path="/var/lib/kubelet/pods/e6b35e72-807a-48b1-a764-9698ffe71dc2/volumes" Oct 30 19:03:16.571777 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:16.571742 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p"] Oct 30 19:03:16.572046 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:16.572014 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" podUID="8085a328-5dcc-44f9-9b91-6933bbbdd301" containerName="istio-proxy" containerID="cri-o://f789b201fd9aab2d1365b951526f56972850d427ce6911fe8c1d69015f341bb9" gracePeriod=30 Oct 30 19:03:16.585439 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:16.585400 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q"] Oct 30 19:03:16.585660 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:16.585638 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" podUID="0e048745-6a7f-4d2f-a9f8-768e7d411370" containerName="istio-proxy" containerID="cri-o://6b96bfb1a78aa999315d4a41893d9acfb38ffdde8f69528a84724a4d8ec50bfb" gracePeriod=30 Oct 30 19:03:16.595354 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:16.595319 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6"] Oct 30 19:03:16.595549 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:16.595528 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" podUID="5a205f4f-e431-45d9-81c4-1f4df8a750ac" containerName="istio-proxy" containerID="cri-o://c423c56d2f9d9ea36cbbd4f0e47a6b6517cb7093b72eaaf3ec23f619ceadac2a" gracePeriod=30 Oct 30 19:03:19.328154 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:19.325305 4531 patch_prober.go:29] interesting pod/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:03:19.328154 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:19.325375 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" podUID="0e048745-6a7f-4d2f-a9f8-768e7d411370" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:03:19.328154 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:19.328072 4531 patch_prober.go:29] interesting pod/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:03:19.328154 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:19.328149 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" podUID="5a205f4f-e431-45d9-81c4-1f4df8a750ac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:03:19.433731 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:19.433676 4531 patch_prober.go:29] interesting pod/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:03:19.433936 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:19.433755 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" podUID="8085a328-5dcc-44f9-9b91-6933bbbdd301" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:03:21.343208 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.343163 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp"] Oct 30 19:03:21.343685 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.343463 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" podUID="90016c74-97f8-4976-87c0-1603b897c2f3" containerName="istio-proxy" containerID="cri-o://4429e7d3ec8c7564c665eeba9841957bb50b148468fcf0c15b77372bd90d04ca" gracePeriod=30 Oct 30 19:03:21.880442 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.880360 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:03:21.936052 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.936024 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:03:21.940015 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.939992 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:03:21.987026 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.986986 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8085a328-5dcc-44f9-9b91-6933bbbdd301-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8085a328-5dcc-44f9-9b91-6933bbbdd301-istiod-ca-cert\") pod \"8085a328-5dcc-44f9-9b91-6933bbbdd301\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " Oct 30 19:03:21.987258 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.987041 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-podinfo\") pod \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " Oct 30 19:03:21.987258 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.987077 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-envoy\") pod \"8085a328-5dcc-44f9-9b91-6933bbbdd301\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " Oct 30 19:03:21.987258 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.987152 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-credential-socket\") pod \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " Oct 30 19:03:21.987258 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.987190 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-workload-certs\") pod \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " Oct 30 19:03:21.987501 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.987476 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-credential-socket" pod "5a205f4f-e431-45d9-81c4-1f4df8a750ac" (UID: "5a205f4f-e431-45d9-81c4-1f4df8a750ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:21.987558 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.987497 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-workload-certs" pod "5a205f4f-e431-45d9-81c4-1f4df8a750ac" (UID: "5a205f4f-e431-45d9-81c4-1f4df8a750ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:21.987609 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.987577 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8085a328-5dcc-44f9-9b91-6933bbbdd301-istiod-ca-cert" pod "8085a328-5dcc-44f9-9b91-6933bbbdd301" (UID: "8085a328-5dcc-44f9-9b91-6933bbbdd301"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:03:21.987787 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.987729 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-envoy\") pod \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " Oct 30 19:03:21.988734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.987874 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-envoy\") pod \"0e048745-6a7f-4d2f-a9f8-768e7d411370\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " Oct 30 19:03:21.988734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.987934 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e048745-6a7f-4d2f-a9f8-768e7d411370-kube-api-access-pqkzg\" (UniqueName: \"kubernetes.io/projected/0e048745-6a7f-4d2f-a9f8-768e7d411370-kube-api-access-pqkzg\") pod \"0e048745-6a7f-4d2f-a9f8-768e7d411370\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " Oct 30 19:03:21.988734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.987965 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-credential-socket\") pod \"0e048745-6a7f-4d2f-a9f8-768e7d411370\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " Oct 30 19:03:21.988734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.987996 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-podinfo\") pod \"0e048745-6a7f-4d2f-a9f8-768e7d411370\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " Oct 30 19:03:21.988734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.988055 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-data\") pod \"0e048745-6a7f-4d2f-a9f8-768e7d411370\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " Oct 30 19:03:21.988734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.988080 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-podinfo\") pod \"8085a328-5dcc-44f9-9b91-6933bbbdd301\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " Oct 30 19:03:21.988734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.988102 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8085a328-5dcc-44f9-9b91-6933bbbdd301-kube-api-access-x7xjg\" (UniqueName: \"kubernetes.io/projected/8085a328-5dcc-44f9-9b91-6933bbbdd301-kube-api-access-x7xjg\") pod \"8085a328-5dcc-44f9-9b91-6933bbbdd301\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " Oct 30 19:03:21.988734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.988158 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a205f4f-e431-45d9-81c4-1f4df8a750ac-kube-api-access-q99jv\" (UniqueName: \"kubernetes.io/projected/5a205f4f-e431-45d9-81c4-1f4df8a750ac-kube-api-access-q99jv\") pod \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " Oct 30 19:03:21.988734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.988213 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-data\") pod \"8085a328-5dcc-44f9-9b91-6933bbbdd301\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " Oct 30 19:03:21.988734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.988255 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-credential-socket\") pod \"8085a328-5dcc-44f9-9b91-6933bbbdd301\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " Oct 30 19:03:21.988734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.988297 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-workload-socket\") pod \"0e048745-6a7f-4d2f-a9f8-768e7d411370\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " Oct 30 19:03:21.988734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.988332 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-workload-certs\") pod \"8085a328-5dcc-44f9-9b91-6933bbbdd301\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " Oct 30 19:03:21.988734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.988364 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-workload-socket\") pod \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " Oct 30 19:03:21.988734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.988397 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e048745-6a7f-4d2f-a9f8-768e7d411370-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e048745-6a7f-4d2f-a9f8-768e7d411370-istiod-ca-cert\") pod \"0e048745-6a7f-4d2f-a9f8-768e7d411370\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " Oct 30 19:03:21.988734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.988459 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-workload-certs\") pod \"0e048745-6a7f-4d2f-a9f8-768e7d411370\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " Oct 30 19:03:21.988734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.988484 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-token\" (UniqueName: \"kubernetes.io/projected/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-token\") pod \"0e048745-6a7f-4d2f-a9f8-768e7d411370\" (UID: \"0e048745-6a7f-4d2f-a9f8-768e7d411370\") " Oct 30 19:03:21.988734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.988509 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istiod-ca-cert\") pod \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " Oct 30 19:03:21.988734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.988537 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-token\" (UniqueName: \"kubernetes.io/projected/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-token\") pod \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " Oct 30 19:03:21.990496 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.989874 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-token\" (UniqueName: \"kubernetes.io/projected/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-token\") pod \"8085a328-5dcc-44f9-9b91-6933bbbdd301\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " Oct 30 19:03:21.990496 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.989938 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-data\") pod \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\" (UID: \"5a205f4f-e431-45d9-81c4-1f4df8a750ac\") " Oct 30 19:03:21.990496 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.989943 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-data" pod "8085a328-5dcc-44f9-9b91-6933bbbdd301" (UID: "8085a328-5dcc-44f9-9b91-6933bbbdd301"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:21.990496 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.989975 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-workload-socket\") pod \"8085a328-5dcc-44f9-9b91-6933bbbdd301\" (UID: \"8085a328-5dcc-44f9-9b91-6933bbbdd301\") " Oct 30 19:03:21.990496 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.990089 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-credential-socket" pod "8085a328-5dcc-44f9-9b91-6933bbbdd301" (UID: "8085a328-5dcc-44f9-9b91-6933bbbdd301"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:21.990496 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.990374 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-workload-socket" pod "0e048745-6a7f-4d2f-a9f8-768e7d411370" (UID: "0e048745-6a7f-4d2f-a9f8-768e7d411370"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:21.990496 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.990477 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-workload-certs" pod "8085a328-5dcc-44f9-9b91-6933bbbdd301" (UID: "8085a328-5dcc-44f9-9b91-6933bbbdd301"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:21.990967 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.990676 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-workload-socket" pod "5a205f4f-e431-45d9-81c4-1f4df8a750ac" (UID: "5a205f4f-e431-45d9-81c4-1f4df8a750ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:21.990967 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.990693 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-data" pod "0e048745-6a7f-4d2f-a9f8-768e7d411370" (UID: "0e048745-6a7f-4d2f-a9f8-768e7d411370"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:21.990967 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.990936 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:21.990967 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.990956 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:21.990967 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.990969 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:21.991278 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.990982 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:21.991278 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.990995 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8085a328-5dcc-44f9-9b91-6933bbbdd301-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:21.991278 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.991000 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-credential-socket" pod "0e048745-6a7f-4d2f-a9f8-768e7d411370" (UID: "0e048745-6a7f-4d2f-a9f8-768e7d411370"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:21.991278 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.991009 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:21.991278 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.991030 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:21.991278 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.991045 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:21.991278 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.991058 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:21.991278 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.991028 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istiod-ca-cert" pod "5a205f4f-e431-45d9-81c4-1f4df8a750ac" (UID: "5a205f4f-e431-45d9-81c4-1f4df8a750ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:03:21.991278 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.991238 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-workload-certs" pod "0e048745-6a7f-4d2f-a9f8-768e7d411370" (UID: "0e048745-6a7f-4d2f-a9f8-768e7d411370"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:21.991735 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.991332 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-data" pod "5a205f4f-e431-45d9-81c4-1f4df8a750ac" (UID: "5a205f4f-e431-45d9-81c4-1f4df8a750ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:21.991735 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.991463 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-workload-socket" pod "8085a328-5dcc-44f9-9b91-6933bbbdd301" (UID: "8085a328-5dcc-44f9-9b91-6933bbbdd301"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:21.991735 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.991543 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e048745-6a7f-4d2f-a9f8-768e7d411370-istiod-ca-cert" pod "0e048745-6a7f-4d2f-a9f8-768e7d411370" (UID: "0e048745-6a7f-4d2f-a9f8-768e7d411370"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:03:21.992046 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.992013 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-envoy" pod "8085a328-5dcc-44f9-9b91-6933bbbdd301" (UID: "8085a328-5dcc-44f9-9b91-6933bbbdd301"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:21.992199 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.992067 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-envoy" pod "5a205f4f-e431-45d9-81c4-1f4df8a750ac" (UID: "5a205f4f-e431-45d9-81c4-1f4df8a750ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:21.993045 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.993016 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-podinfo" pod "0e048745-6a7f-4d2f-a9f8-768e7d411370" (UID: "0e048745-6a7f-4d2f-a9f8-768e7d411370"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:03:21.993483 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.993455 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-podinfo" pod "5a205f4f-e431-45d9-81c4-1f4df8a750ac" (UID: "5a205f4f-e431-45d9-81c4-1f4df8a750ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:03:21.994032 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.993987 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-podinfo" pod "8085a328-5dcc-44f9-9b91-6933bbbdd301" (UID: "8085a328-5dcc-44f9-9b91-6933bbbdd301"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:03:21.994280 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.994259 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-token" pod "8085a328-5dcc-44f9-9b91-6933bbbdd301" (UID: "8085a328-5dcc-44f9-9b91-6933bbbdd301"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:03:21.994361 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.994293 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e048745-6a7f-4d2f-a9f8-768e7d411370-kube-api-access-pqkzg" pod "0e048745-6a7f-4d2f-a9f8-768e7d411370" (UID: "0e048745-6a7f-4d2f-a9f8-768e7d411370"). InnerVolumeSpecName "kube-api-access-pqkzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:03:21.994361 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.994336 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-token" pod "5a205f4f-e431-45d9-81c4-1f4df8a750ac" (UID: "5a205f4f-e431-45d9-81c4-1f4df8a750ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:03:21.994676 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.994651 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8085a328-5dcc-44f9-9b91-6933bbbdd301-kube-api-access-x7xjg" pod "8085a328-5dcc-44f9-9b91-6933bbbdd301" (UID: "8085a328-5dcc-44f9-9b91-6933bbbdd301"). InnerVolumeSpecName "kube-api-access-x7xjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:03:21.994873 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.994849 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-envoy" pod "0e048745-6a7f-4d2f-a9f8-768e7d411370" (UID: "0e048745-6a7f-4d2f-a9f8-768e7d411370"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:21.994873 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.994856 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a205f4f-e431-45d9-81c4-1f4df8a750ac-kube-api-access-q99jv" pod "5a205f4f-e431-45d9-81c4-1f4df8a750ac" (UID: "5a205f4f-e431-45d9-81c4-1f4df8a750ac"). InnerVolumeSpecName "kube-api-access-q99jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:03:21.995101 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:21.995083 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-token" pod "0e048745-6a7f-4d2f-a9f8-768e7d411370" (UID: "0e048745-6a7f-4d2f-a9f8-768e7d411370"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:03:22.092050 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.092014 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:22.092050 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.092047 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:22.092331 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.092060 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:22.092331 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.092073 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:22.092331 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.092084 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:22.092331 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.092096 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:22.092331 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.092106 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:22.092331 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.092118 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:22.092331 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.092166 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:22.092331 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.092186 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a205f4f-e431-45d9-81c4-1f4df8a750ac-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:22.092331 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.092197 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:22.092331 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.092210 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqkzg\" (UniqueName: \"kubernetes.io/projected/0e048745-6a7f-4d2f-a9f8-768e7d411370-kube-api-access-pqkzg\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:22.092331 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.092227 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e048745-6a7f-4d2f-a9f8-768e7d411370-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:22.092331 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.092239 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e048745-6a7f-4d2f-a9f8-768e7d411370-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:22.092331 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.092250 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8085a328-5dcc-44f9-9b91-6933bbbdd301-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:22.092331 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.092262 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7xjg\" (UniqueName: \"kubernetes.io/projected/8085a328-5dcc-44f9-9b91-6933bbbdd301-kube-api-access-x7xjg\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:22.092331 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.092281 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q99jv\" (UniqueName: \"kubernetes.io/projected/5a205f4f-e431-45d9-81c4-1f4df8a750ac-kube-api-access-q99jv\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:22.092331 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.092294 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e048745-6a7f-4d2f-a9f8-768e7d411370-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:22.218579 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.218542 4531 generic.go:356] "Generic (PLEG): container finished" podID="8085a328-5dcc-44f9-9b91-6933bbbdd301" containerID="f789b201fd9aab2d1365b951526f56972850d427ce6911fe8c1d69015f341bb9" exitCode=0 Oct 30 19:03:22.218739 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.218615 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" Oct 30 19:03:22.218739 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.218612 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" event={"ID":"8085a328-5dcc-44f9-9b91-6933bbbdd301","Type":"ContainerDied","Data":"f789b201fd9aab2d1365b951526f56972850d427ce6911fe8c1d69015f341bb9"} Oct 30 19:03:22.218872 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.218819 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p" event={"ID":"8085a328-5dcc-44f9-9b91-6933bbbdd301","Type":"ContainerDied","Data":"d24fc2dcebd2c3e04383881a2a7521307ed744aaff83fd74779069b95b88e6e7"} Oct 30 19:03:22.218872 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.218857 4531 scope.go:122] "RemoveContainer" containerID="f789b201fd9aab2d1365b951526f56972850d427ce6911fe8c1d69015f341bb9" Oct 30 19:03:22.220214 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.220188 4531 generic.go:356] "Generic (PLEG): container finished" podID="0e048745-6a7f-4d2f-a9f8-768e7d411370" containerID="6b96bfb1a78aa999315d4a41893d9acfb38ffdde8f69528a84724a4d8ec50bfb" exitCode=0 Oct 30 19:03:22.220340 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.220246 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" event={"ID":"0e048745-6a7f-4d2f-a9f8-768e7d411370","Type":"ContainerDied","Data":"6b96bfb1a78aa999315d4a41893d9acfb38ffdde8f69528a84724a4d8ec50bfb"} Oct 30 19:03:22.220340 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.220306 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" event={"ID":"0e048745-6a7f-4d2f-a9f8-768e7d411370","Type":"ContainerDied","Data":"9a1beb1ed9abe38c6897e6019c14cc225a86e257fda71221730f6947b6fed578"} Oct 30 19:03:22.220340 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.220254 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q" Oct 30 19:03:22.222112 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.222085 4531 generic.go:356] "Generic (PLEG): container finished" podID="5a205f4f-e431-45d9-81c4-1f4df8a750ac" containerID="c423c56d2f9d9ea36cbbd4f0e47a6b6517cb7093b72eaaf3ec23f619ceadac2a" exitCode=0 Oct 30 19:03:22.222222 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.222155 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" event={"ID":"5a205f4f-e431-45d9-81c4-1f4df8a750ac","Type":"ContainerDied","Data":"c423c56d2f9d9ea36cbbd4f0e47a6b6517cb7093b72eaaf3ec23f619ceadac2a"} Oct 30 19:03:22.222222 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.222173 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" Oct 30 19:03:22.222222 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.222195 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6" event={"ID":"5a205f4f-e431-45d9-81c4-1f4df8a750ac","Type":"ContainerDied","Data":"17b350627ca94524a0289e6cdcb3730b03b96f8d9fc50d8815f5064a3a5d4a78"} Oct 30 19:03:22.232968 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.232926 4531 scope.go:122] "RemoveContainer" containerID="f789b201fd9aab2d1365b951526f56972850d427ce6911fe8c1d69015f341bb9" Oct 30 19:03:22.233393 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:03:22.233347 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f789b201fd9aab2d1365b951526f56972850d427ce6911fe8c1d69015f341bb9\": container with ID starting with f789b201fd9aab2d1365b951526f56972850d427ce6911fe8c1d69015f341bb9 not found: ID does not exist" containerID="f789b201fd9aab2d1365b951526f56972850d427ce6911fe8c1d69015f341bb9" Oct 30 19:03:22.233517 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.233395 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f789b201fd9aab2d1365b951526f56972850d427ce6911fe8c1d69015f341bb9"} err="failed to get container status \"f789b201fd9aab2d1365b951526f56972850d427ce6911fe8c1d69015f341bb9\": rpc error: code = NotFound desc = could not find container \"f789b201fd9aab2d1365b951526f56972850d427ce6911fe8c1d69015f341bb9\": container with ID starting with f789b201fd9aab2d1365b951526f56972850d427ce6911fe8c1d69015f341bb9 not found: ID does not exist" Oct 30 19:03:22.233517 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.233421 4531 scope.go:122] "RemoveContainer" containerID="6b96bfb1a78aa999315d4a41893d9acfb38ffdde8f69528a84724a4d8ec50bfb" Oct 30 19:03:22.245438 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.245340 4531 scope.go:122] "RemoveContainer" containerID="6b96bfb1a78aa999315d4a41893d9acfb38ffdde8f69528a84724a4d8ec50bfb" Oct 30 19:03:22.245819 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:03:22.245790 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b96bfb1a78aa999315d4a41893d9acfb38ffdde8f69528a84724a4d8ec50bfb\": container with ID starting with 6b96bfb1a78aa999315d4a41893d9acfb38ffdde8f69528a84724a4d8ec50bfb not found: ID does not exist" containerID="6b96bfb1a78aa999315d4a41893d9acfb38ffdde8f69528a84724a4d8ec50bfb" Oct 30 19:03:22.245936 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.245832 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b96bfb1a78aa999315d4a41893d9acfb38ffdde8f69528a84724a4d8ec50bfb"} err="failed to get container status \"6b96bfb1a78aa999315d4a41893d9acfb38ffdde8f69528a84724a4d8ec50bfb\": rpc error: code = NotFound desc = could not find container \"6b96bfb1a78aa999315d4a41893d9acfb38ffdde8f69528a84724a4d8ec50bfb\": container with ID starting with 6b96bfb1a78aa999315d4a41893d9acfb38ffdde8f69528a84724a4d8ec50bfb not found: ID does not exist" Oct 30 19:03:22.245936 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.245857 4531 scope.go:122] "RemoveContainer" containerID="c423c56d2f9d9ea36cbbd4f0e47a6b6517cb7093b72eaaf3ec23f619ceadac2a" Oct 30 19:03:22.259213 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.259187 4531 scope.go:122] "RemoveContainer" containerID="c423c56d2f9d9ea36cbbd4f0e47a6b6517cb7093b72eaaf3ec23f619ceadac2a" Oct 30 19:03:22.259644 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:03:22.259614 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c423c56d2f9d9ea36cbbd4f0e47a6b6517cb7093b72eaaf3ec23f619ceadac2a\": container with ID starting with c423c56d2f9d9ea36cbbd4f0e47a6b6517cb7093b72eaaf3ec23f619ceadac2a not found: ID does not exist" containerID="c423c56d2f9d9ea36cbbd4f0e47a6b6517cb7093b72eaaf3ec23f619ceadac2a" Oct 30 19:03:22.259771 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.259652 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c423c56d2f9d9ea36cbbd4f0e47a6b6517cb7093b72eaaf3ec23f619ceadac2a"} err="failed to get container status \"c423c56d2f9d9ea36cbbd4f0e47a6b6517cb7093b72eaaf3ec23f619ceadac2a\": rpc error: code = NotFound desc = could not find container \"c423c56d2f9d9ea36cbbd4f0e47a6b6517cb7093b72eaaf3ec23f619ceadac2a\": container with ID starting with c423c56d2f9d9ea36cbbd4f0e47a6b6517cb7093b72eaaf3ec23f619ceadac2a not found: ID does not exist" Oct 30 19:03:22.381386 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.381342 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6"] Oct 30 19:03:22.386721 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.386681 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-qppq6"] Oct 30 19:03:22.507162 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.502330 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q"] Oct 30 19:03:22.507162 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.503035 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-86s8q"] Oct 30 19:03:22.632294 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.632257 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p"] Oct 30 19:03:22.646163 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:22.646112 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-66baf45c-openshift-default-974f9d7b9-mp48p"] Oct 30 19:03:23.678492 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.678410 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q"] Oct 30 19:03:23.680405 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.679925 4531 cpu_manager.go:414] "containerMap: removing container" podUID="88fc33e1-96f8-4682-aedf-ef669f85138c" containerName="istio-proxy" Oct 30 19:03:23.681290 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.681251 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88fc33e1-96f8-4682-aedf-ef669f85138c" containerName="istio-proxy" Oct 30 19:03:23.682423 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.682399 4531 cpu_manager.go:414] "containerMap: removing container" podUID="2b2ed0c4-859a-41eb-9391-8720e3b58365" containerName="revoke-keys" Oct 30 19:03:23.682593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.682573 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b2ed0c4-859a-41eb-9391-8720e3b58365" containerName="revoke-keys" Oct 30 19:03:23.682666 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.682603 4531 cpu_manager.go:414] "containerMap: removing container" podUID="dab47e85-278b-4d4f-94b2-8f46ec744d69" containerName="main" Oct 30 19:03:23.682666 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.682614 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dab47e85-278b-4d4f-94b2-8f46ec744d69" containerName="main" Oct 30 19:03:23.682666 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.682628 4531 cpu_manager.go:414] "containerMap: removing container" podUID="5a205f4f-e431-45d9-81c4-1f4df8a750ac" containerName="istio-proxy" Oct 30 19:03:23.682666 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.682636 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a205f4f-e431-45d9-81c4-1f4df8a750ac" containerName="istio-proxy" Oct 30 19:03:23.682666 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.682650 4531 cpu_manager.go:414] "containerMap: removing container" podUID="0e048745-6a7f-4d2f-a9f8-768e7d411370" containerName="istio-proxy" Oct 30 19:03:23.682666 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.682660 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e048745-6a7f-4d2f-a9f8-768e7d411370" containerName="istio-proxy" Oct 30 19:03:23.682946 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.682674 4531 cpu_manager.go:414] "containerMap: removing container" podUID="6f8455f6-97f8-47c7-b862-34d0a948329b" containerName="payload-pre-processing" Oct 30 19:03:23.682946 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.682682 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f8455f6-97f8-47c7-b862-34d0a948329b" containerName="payload-pre-processing" Oct 30 19:03:23.682946 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.682697 4531 cpu_manager.go:414] "containerMap: removing container" podUID="8085a328-5dcc-44f9-9b91-6933bbbdd301" containerName="istio-proxy" Oct 30 19:03:23.682946 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.682705 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8085a328-5dcc-44f9-9b91-6933bbbdd301" containerName="istio-proxy" Oct 30 19:03:23.683121 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.683001 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="88fc33e1-96f8-4682-aedf-ef669f85138c" containerName="istio-proxy" Oct 30 19:03:23.683121 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.683016 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b2ed0c4-859a-41eb-9391-8720e3b58365" containerName="revoke-keys" Oct 30 19:03:23.683121 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.683043 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="dab47e85-278b-4d4f-94b2-8f46ec744d69" containerName="main" Oct 30 19:03:23.683121 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.683054 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a205f4f-e431-45d9-81c4-1f4df8a750ac" containerName="istio-proxy" Oct 30 19:03:23.683121 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.683064 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e048745-6a7f-4d2f-a9f8-768e7d411370" containerName="istio-proxy" Oct 30 19:03:23.683121 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.683072 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f8455f6-97f8-47c7-b862-34d0a948329b" containerName="payload-pre-processing" Oct 30 19:03:23.683121 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.683081 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8085a328-5dcc-44f9-9b91-6933bbbdd301" containerName="istio-proxy" Oct 30 19:03:23.694163 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.688734 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q" Oct 30 19:03:23.740991 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.740913 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q"] Oct 30 19:03:23.742694 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.742660 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:03:23.786604 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.786558 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:03:23.786604 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.786562 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:03:23.786604 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.786589 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:03:23.862751 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.862691 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58ca32e1-9bcb-4600-b2ad-17e3656c4db6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q\" (UID: \"58ca32e1-9bcb-4600-b2ad-17e3656c4db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q" Oct 30 19:03:23.862929 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.862900 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9l7x\" (UniqueName: \"kubernetes.io/projected/58ca32e1-9bcb-4600-b2ad-17e3656c4db6-kube-api-access-q9l7x\") pod \"maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q\" (UID: \"58ca32e1-9bcb-4600-b2ad-17e3656c4db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q" Oct 30 19:03:23.863027 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.863014 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58ca32e1-9bcb-4600-b2ad-17e3656c4db6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q\" (UID: \"58ca32e1-9bcb-4600-b2ad-17e3656c4db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q" Oct 30 19:03:23.964043 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.963938 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58ca32e1-9bcb-4600-b2ad-17e3656c4db6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q\" (UID: \"58ca32e1-9bcb-4600-b2ad-17e3656c4db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q" Oct 30 19:03:23.964273 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.964063 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9l7x\" (UniqueName: \"kubernetes.io/projected/58ca32e1-9bcb-4600-b2ad-17e3656c4db6-kube-api-access-q9l7x\") pod \"maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q\" (UID: \"58ca32e1-9bcb-4600-b2ad-17e3656c4db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q" Oct 30 19:03:23.964273 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.964170 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58ca32e1-9bcb-4600-b2ad-17e3656c4db6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q\" (UID: \"58ca32e1-9bcb-4600-b2ad-17e3656c4db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q" Oct 30 19:03:23.964731 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.964703 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58ca32e1-9bcb-4600-b2ad-17e3656c4db6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q\" (UID: \"58ca32e1-9bcb-4600-b2ad-17e3656c4db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q" Oct 30 19:03:23.966877 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:23.966852 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58ca32e1-9bcb-4600-b2ad-17e3656c4db6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q\" (UID: \"58ca32e1-9bcb-4600-b2ad-17e3656c4db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q" Oct 30 19:03:24.025426 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:24.025394 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9l7x\" (UniqueName: \"kubernetes.io/projected/58ca32e1-9bcb-4600-b2ad-17e3656c4db6-kube-api-access-q9l7x\") pod \"maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q\" (UID: \"58ca32e1-9bcb-4600-b2ad-17e3656c4db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q" Oct 30 19:03:24.099681 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:24.099644 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e048745-6a7f-4d2f-a9f8-768e7d411370" path="/var/lib/kubelet/pods/0e048745-6a7f-4d2f-a9f8-768e7d411370/volumes" Oct 30 19:03:24.100186 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:24.100169 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a205f4f-e431-45d9-81c4-1f4df8a750ac" path="/var/lib/kubelet/pods/5a205f4f-e431-45d9-81c4-1f4df8a750ac/volumes" Oct 30 19:03:24.100630 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:24.100614 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8085a328-5dcc-44f9-9b91-6933bbbdd301" path="/var/lib/kubelet/pods/8085a328-5dcc-44f9-9b91-6933bbbdd301/volumes" Oct 30 19:03:24.311689 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:24.311640 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q" Oct 30 19:03:24.702705 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:24.702664 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q"] Oct 30 19:03:24.704358 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:03:24.704323 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ca32e1_9bcb_4600_b2ad_17e3656c4db6.slice/crio-146b0e8b79a57a6e82cdf355c36c28cbe4028e3e4a8e4bbe61961a683adfee5e WatchSource:0}: Error finding container 146b0e8b79a57a6e82cdf355c36c28cbe4028e3e4a8e4bbe61961a683adfee5e: Status 404 returned error can't find the container with id 146b0e8b79a57a6e82cdf355c36c28cbe4028e3e4a8e4bbe61961a683adfee5e Oct 30 19:03:25.268038 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:25.267989 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q" event={"ID":"58ca32e1-9bcb-4600-b2ad-17e3656c4db6","Type":"ContainerStarted","Data":"677bca08eefe3781db91acc358fb8aca2766ff6a0604c7f33d4c9e409987d2d4"} Oct 30 19:03:25.268240 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:25.268097 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q" event={"ID":"58ca32e1-9bcb-4600-b2ad-17e3656c4db6","Type":"ContainerStarted","Data":"146b0e8b79a57a6e82cdf355c36c28cbe4028e3e4a8e4bbe61961a683adfee5e"} Oct 30 19:03:25.401652 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:25.401528 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q" podStartSLOduration=2.401512306 podStartE2EDuration="2.401512306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:25.399322027 +0000 UTC m=+1765.426426978" watchObservedRunningTime="2026-10-30 19:03:25.401512306 +0000 UTC m=+1765.428617264" Oct 30 19:03:25.533925 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:25.533884 4531 patch_prober.go:29] interesting pod/e2e-api-b-470093-openshift-default-6745c4765-8xfqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:03:25.534098 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:25.533959 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" podUID="90016c74-97f8-4976-87c0-1603b897c2f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:03:26.276739 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:26.276697 4531 generic.go:356] "Generic (PLEG): container finished" podID="58ca32e1-9bcb-4600-b2ad-17e3656c4db6" containerID="677bca08eefe3781db91acc358fb8aca2766ff6a0604c7f33d4c9e409987d2d4" exitCode=0 Oct 30 19:03:26.277177 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:26.276768 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q" event={"ID":"58ca32e1-9bcb-4600-b2ad-17e3656c4db6","Type":"ContainerDied","Data":"677bca08eefe3781db91acc358fb8aca2766ff6a0604c7f33d4c9e409987d2d4"} Oct 30 19:03:26.402993 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:03:26.402949 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:03:27.137362 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.137333 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 19:03:27.246242 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.246205 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-workload-socket\") pod \"90016c74-97f8-4976-87c0-1603b897c2f3\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " Oct 30 19:03:27.246426 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.246254 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-istio-envoy\") pod \"90016c74-97f8-4976-87c0-1603b897c2f3\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " Oct 30 19:03:27.246426 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.246301 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90016c74-97f8-4976-87c0-1603b897c2f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90016c74-97f8-4976-87c0-1603b897c2f3-istiod-ca-cert\") pod \"90016c74-97f8-4976-87c0-1603b897c2f3\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " Oct 30 19:03:27.246426 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.246335 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-credential-socket\") pod \"90016c74-97f8-4976-87c0-1603b897c2f3\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " Oct 30 19:03:27.246426 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.246361 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90016c74-97f8-4976-87c0-1603b897c2f3-kube-api-access-7hkw8\" (UniqueName: \"kubernetes.io/projected/90016c74-97f8-4976-87c0-1603b897c2f3-kube-api-access-7hkw8\") pod \"90016c74-97f8-4976-87c0-1603b897c2f3\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " Oct 30 19:03:27.246426 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.246394 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-workload-certs\") pod \"90016c74-97f8-4976-87c0-1603b897c2f3\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " Oct 30 19:03:27.246667 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.246477 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-istio-data\") pod \"90016c74-97f8-4976-87c0-1603b897c2f3\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " Oct 30 19:03:27.246667 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.246520 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90016c74-97f8-4976-87c0-1603b897c2f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90016c74-97f8-4976-87c0-1603b897c2f3-istio-podinfo\") pod \"90016c74-97f8-4976-87c0-1603b897c2f3\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " Oct 30 19:03:27.246667 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.246560 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-workload-socket" pod "90016c74-97f8-4976-87c0-1603b897c2f3" (UID: "90016c74-97f8-4976-87c0-1603b897c2f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:27.246667 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.246620 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90016c74-97f8-4976-87c0-1603b897c2f3-istio-token\" (UniqueName: \"kubernetes.io/projected/90016c74-97f8-4976-87c0-1603b897c2f3-istio-token\") pod \"90016c74-97f8-4976-87c0-1603b897c2f3\" (UID: \"90016c74-97f8-4976-87c0-1603b897c2f3\") " Oct 30 19:03:27.246875 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.246780 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-istio-data" pod "90016c74-97f8-4976-87c0-1603b897c2f3" (UID: "90016c74-97f8-4976-87c0-1603b897c2f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:27.246875 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.246795 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-workload-certs" pod "90016c74-97f8-4976-87c0-1603b897c2f3" (UID: "90016c74-97f8-4976-87c0-1603b897c2f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:27.246875 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.246799 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90016c74-97f8-4976-87c0-1603b897c2f3-istiod-ca-cert" pod "90016c74-97f8-4976-87c0-1603b897c2f3" (UID: "90016c74-97f8-4976-87c0-1603b897c2f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:03:27.247171 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.247119 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-credential-socket" pod "90016c74-97f8-4976-87c0-1603b897c2f3" (UID: "90016c74-97f8-4976-87c0-1603b897c2f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:27.247171 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.247157 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:27.247334 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.247182 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:27.247334 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.247196 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90016c74-97f8-4976-87c0-1603b897c2f3-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:27.247334 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.247209 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:27.248776 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.248751 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-istio-envoy" pod "90016c74-97f8-4976-87c0-1603b897c2f3" (UID: "90016c74-97f8-4976-87c0-1603b897c2f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:27.248950 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.248935 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90016c74-97f8-4976-87c0-1603b897c2f3-kube-api-access-7hkw8" pod "90016c74-97f8-4976-87c0-1603b897c2f3" (UID: "90016c74-97f8-4976-87c0-1603b897c2f3"). InnerVolumeSpecName "kube-api-access-7hkw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:03:27.249412 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.249395 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90016c74-97f8-4976-87c0-1603b897c2f3-istio-token" pod "90016c74-97f8-4976-87c0-1603b897c2f3" (UID: "90016c74-97f8-4976-87c0-1603b897c2f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:03:27.249412 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.249395 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90016c74-97f8-4976-87c0-1603b897c2f3-istio-podinfo" pod "90016c74-97f8-4976-87c0-1603b897c2f3" (UID: "90016c74-97f8-4976-87c0-1603b897c2f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:03:27.288058 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.288009 4531 generic.go:356] "Generic (PLEG): container finished" podID="90016c74-97f8-4976-87c0-1603b897c2f3" containerID="4429e7d3ec8c7564c665eeba9841957bb50b148468fcf0c15b77372bd90d04ca" exitCode=0 Oct 30 19:03:27.288601 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.288068 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" Oct 30 19:03:27.288601 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.288094 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" event={"ID":"90016c74-97f8-4976-87c0-1603b897c2f3","Type":"ContainerDied","Data":"4429e7d3ec8c7564c665eeba9841957bb50b148468fcf0c15b77372bd90d04ca"} Oct 30 19:03:27.288601 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.288187 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp" event={"ID":"90016c74-97f8-4976-87c0-1603b897c2f3","Type":"ContainerDied","Data":"2a3a1f28ad651e8be5a7795c96621728c3b6bcfbd45b0b452e89e0134951d21d"} Oct 30 19:03:27.288601 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.288221 4531 scope.go:122] "RemoveContainer" containerID="4429e7d3ec8c7564c665eeba9841957bb50b148468fcf0c15b77372bd90d04ca" Oct 30 19:03:27.302791 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.302761 4531 scope.go:122] "RemoveContainer" containerID="595c6a03ff16516ae450f50452663d04111f911a3c4c9d0a5f4586f5f88f9c14" Oct 30 19:03:27.315112 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.315089 4531 scope.go:122] "RemoveContainer" containerID="4429e7d3ec8c7564c665eeba9841957bb50b148468fcf0c15b77372bd90d04ca" Oct 30 19:03:27.315511 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:03:27.315481 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4429e7d3ec8c7564c665eeba9841957bb50b148468fcf0c15b77372bd90d04ca\": container with ID starting with 4429e7d3ec8c7564c665eeba9841957bb50b148468fcf0c15b77372bd90d04ca not found: ID does not exist" containerID="4429e7d3ec8c7564c665eeba9841957bb50b148468fcf0c15b77372bd90d04ca" Oct 30 19:03:27.315601 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.315527 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4429e7d3ec8c7564c665eeba9841957bb50b148468fcf0c15b77372bd90d04ca"} err="failed to get container status \"4429e7d3ec8c7564c665eeba9841957bb50b148468fcf0c15b77372bd90d04ca\": rpc error: code = NotFound desc = could not find container \"4429e7d3ec8c7564c665eeba9841957bb50b148468fcf0c15b77372bd90d04ca\": container with ID starting with 4429e7d3ec8c7564c665eeba9841957bb50b148468fcf0c15b77372bd90d04ca not found: ID does not exist" Oct 30 19:03:27.315601 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.315559 4531 scope.go:122] "RemoveContainer" containerID="595c6a03ff16516ae450f50452663d04111f911a3c4c9d0a5f4586f5f88f9c14" Oct 30 19:03:27.315836 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:03:27.315818 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"595c6a03ff16516ae450f50452663d04111f911a3c4c9d0a5f4586f5f88f9c14\": container with ID starting with 595c6a03ff16516ae450f50452663d04111f911a3c4c9d0a5f4586f5f88f9c14 not found: ID does not exist" containerID="595c6a03ff16516ae450f50452663d04111f911a3c4c9d0a5f4586f5f88f9c14" Oct 30 19:03:27.315904 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.315844 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"595c6a03ff16516ae450f50452663d04111f911a3c4c9d0a5f4586f5f88f9c14"} err="failed to get container status \"595c6a03ff16516ae450f50452663d04111f911a3c4c9d0a5f4586f5f88f9c14\": rpc error: code = NotFound desc = could not find container \"595c6a03ff16516ae450f50452663d04111f911a3c4c9d0a5f4586f5f88f9c14\": container with ID starting with 595c6a03ff16516ae450f50452663d04111f911a3c4c9d0a5f4586f5f88f9c14 not found: ID does not exist" Oct 30 19:03:27.348489 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.348452 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:27.348489 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.348485 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hkw8\" (UniqueName: \"kubernetes.io/projected/90016c74-97f8-4976-87c0-1603b897c2f3-kube-api-access-7hkw8\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:27.348489 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.348495 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90016c74-97f8-4976-87c0-1603b897c2f3-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:27.348720 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.348505 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90016c74-97f8-4976-87c0-1603b897c2f3-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:27.348720 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.348514 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90016c74-97f8-4976-87c0-1603b897c2f3-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:27.453908 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.453876 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp"] Oct 30 19:03:27.459298 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.459261 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-470093-openshift-default-6745c4765-8xfqp"] Oct 30 19:03:27.462654 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.462624 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q" Oct 30 19:03:27.550667 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.550630 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58ca32e1-9bcb-4600-b2ad-17e3656c4db6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58ca32e1-9bcb-4600-b2ad-17e3656c4db6-maas-api-xks-ca\") pod \"58ca32e1-9bcb-4600-b2ad-17e3656c4db6\" (UID: \"58ca32e1-9bcb-4600-b2ad-17e3656c4db6\") " Oct 30 19:03:27.550859 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.550725 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58ca32e1-9bcb-4600-b2ad-17e3656c4db6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58ca32e1-9bcb-4600-b2ad-17e3656c4db6-maas-api-openshift-ca\") pod \"58ca32e1-9bcb-4600-b2ad-17e3656c4db6\" (UID: \"58ca32e1-9bcb-4600-b2ad-17e3656c4db6\") " Oct 30 19:03:27.550859 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.550790 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58ca32e1-9bcb-4600-b2ad-17e3656c4db6-kube-api-access-q9l7x\" (UniqueName: \"kubernetes.io/projected/58ca32e1-9bcb-4600-b2ad-17e3656c4db6-kube-api-access-q9l7x\") pod \"58ca32e1-9bcb-4600-b2ad-17e3656c4db6\" (UID: \"58ca32e1-9bcb-4600-b2ad-17e3656c4db6\") " Oct 30 19:03:27.551091 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.551063 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ca32e1-9bcb-4600-b2ad-17e3656c4db6-maas-api-openshift-ca" pod "58ca32e1-9bcb-4600-b2ad-17e3656c4db6" (UID: "58ca32e1-9bcb-4600-b2ad-17e3656c4db6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:03:27.551407 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.551389 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58ca32e1-9bcb-4600-b2ad-17e3656c4db6-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:27.553199 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.553175 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58ca32e1-9bcb-4600-b2ad-17e3656c4db6-maas-api-xks-ca" pod "58ca32e1-9bcb-4600-b2ad-17e3656c4db6" (UID: "58ca32e1-9bcb-4600-b2ad-17e3656c4db6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:03:27.553324 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.553299 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ca32e1-9bcb-4600-b2ad-17e3656c4db6-kube-api-access-q9l7x" pod "58ca32e1-9bcb-4600-b2ad-17e3656c4db6" (UID: "58ca32e1-9bcb-4600-b2ad-17e3656c4db6"). InnerVolumeSpecName "kube-api-access-q9l7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:03:27.652959 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.652915 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58ca32e1-9bcb-4600-b2ad-17e3656c4db6-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:27.652959 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:27.652955 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9l7x\" (UniqueName: \"kubernetes.io/projected/58ca32e1-9bcb-4600-b2ad-17e3656c4db6-kube-api-access-q9l7x\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:28.101206 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:28.101154 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90016c74-97f8-4976-87c0-1603b897c2f3" path="/var/lib/kubelet/pods/90016c74-97f8-4976-87c0-1603b897c2f3/volumes" Oct 30 19:03:28.298621 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:28.298593 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q" Oct 30 19:03:28.299019 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:28.298628 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q" event={"ID":"58ca32e1-9bcb-4600-b2ad-17e3656c4db6","Type":"ContainerDied","Data":"146b0e8b79a57a6e82cdf355c36c28cbe4028e3e4a8e4bbe61961a683adfee5e"} Oct 30 19:03:28.299019 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:28.298676 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="146b0e8b79a57a6e82cdf355c36c28cbe4028e3e4a8e4bbe61961a683adfee5e" Oct 30 19:03:35.052512 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:35.052470 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc"] Oct 30 19:03:35.168632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:35.168592 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm"] Oct 30 19:03:38.052778 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:38.052733 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2730df4c-5q8g7"] Oct 30 19:03:38.101695 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:38.101662 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2730df4c-5q8g7"] Oct 30 19:03:40.053679 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.053627 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc" podUID="dfe944db-5201-4af3-8bb1-8e616ff90b00" containerName="payload-processing" containerID="cri-o://41c019b4d2660170d997906c1f34cbd78f55ce092484b9b69193a8cc8c5f49c2" gracePeriod=25 Oct 30 19:03:40.100335 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.100300 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f9e1a62-b37c-476a-9d84-586b373579ae" path="/var/lib/kubelet/pods/3f9e1a62-b37c-476a-9d84-586b373579ae/volumes" Oct 30 19:03:40.169239 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.169171 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm" podUID="c3aece38-123d-42a2-a192-298b1d6b1579" containerName="payload-pre-processing" containerID="cri-o://ef5eb488805d44a6b4fc4e568334eebf696f1319328db5e302f929bb39fc9d8e" gracePeriod=25 Oct 30 19:03:40.334028 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.333994 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc" Oct 30 19:03:40.386817 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.386766 4531 generic.go:356] "Generic (PLEG): container finished" podID="dfe944db-5201-4af3-8bb1-8e616ff90b00" containerID="41c019b4d2660170d997906c1f34cbd78f55ce092484b9b69193a8cc8c5f49c2" exitCode=0 Oct 30 19:03:40.386817 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.386803 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc" Oct 30 19:03:40.387073 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.386836 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc" event={"ID":"dfe944db-5201-4af3-8bb1-8e616ff90b00","Type":"ContainerDied","Data":"41c019b4d2660170d997906c1f34cbd78f55ce092484b9b69193a8cc8c5f49c2"} Oct 30 19:03:40.387073 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.386896 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc" event={"ID":"dfe944db-5201-4af3-8bb1-8e616ff90b00","Type":"ContainerDied","Data":"88c91e977a055c427db89be11ce1327941e8ca2c48527e7b4b0dcbac57e7570d"} Oct 30 19:03:40.387073 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.386929 4531 scope.go:122] "RemoveContainer" containerID="41c019b4d2660170d997906c1f34cbd78f55ce092484b9b69193a8cc8c5f49c2" Oct 30 19:03:40.389838 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.389813 4531 generic.go:356] "Generic (PLEG): container finished" podID="c3aece38-123d-42a2-a192-298b1d6b1579" containerID="ef5eb488805d44a6b4fc4e568334eebf696f1319328db5e302f929bb39fc9d8e" exitCode=0 Oct 30 19:03:40.390045 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.389860 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm" event={"ID":"c3aece38-123d-42a2-a192-298b1d6b1579","Type":"ContainerDied","Data":"ef5eb488805d44a6b4fc4e568334eebf696f1319328db5e302f929bb39fc9d8e"} Oct 30 19:03:40.400430 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.400399 4531 scope.go:122] "RemoveContainer" containerID="41c019b4d2660170d997906c1f34cbd78f55ce092484b9b69193a8cc8c5f49c2" Oct 30 19:03:40.400817 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:03:40.400792 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41c019b4d2660170d997906c1f34cbd78f55ce092484b9b69193a8cc8c5f49c2\": container with ID starting with 41c019b4d2660170d997906c1f34cbd78f55ce092484b9b69193a8cc8c5f49c2 not found: ID does not exist" containerID="41c019b4d2660170d997906c1f34cbd78f55ce092484b9b69193a8cc8c5f49c2" Oct 30 19:03:40.400913 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.400829 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41c019b4d2660170d997906c1f34cbd78f55ce092484b9b69193a8cc8c5f49c2"} err="failed to get container status \"41c019b4d2660170d997906c1f34cbd78f55ce092484b9b69193a8cc8c5f49c2\": rpc error: code = NotFound desc = could not find container \"41c019b4d2660170d997906c1f34cbd78f55ce092484b9b69193a8cc8c5f49c2\": container with ID starting with 41c019b4d2660170d997906c1f34cbd78f55ce092484b9b69193a8cc8c5f49c2 not found: ID does not exist" Oct 30 19:03:40.442558 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.442534 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm" Oct 30 19:03:40.448637 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.448614 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfe944db-5201-4af3-8bb1-8e616ff90b00-kube-api-access-h6jg7\" (UniqueName: \"kubernetes.io/projected/dfe944db-5201-4af3-8bb1-8e616ff90b00-kube-api-access-h6jg7\") pod \"dfe944db-5201-4af3-8bb1-8e616ff90b00\" (UID: \"dfe944db-5201-4af3-8bb1-8e616ff90b00\") " Oct 30 19:03:40.448793 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.448779 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfe944db-5201-4af3-8bb1-8e616ff90b00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfe944db-5201-4af3-8bb1-8e616ff90b00-plugins-config-volume\") pod \"dfe944db-5201-4af3-8bb1-8e616ff90b00\" (UID: \"dfe944db-5201-4af3-8bb1-8e616ff90b00\") " Oct 30 19:03:40.449174 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.449115 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfe944db-5201-4af3-8bb1-8e616ff90b00-plugins-config-volume" pod "dfe944db-5201-4af3-8bb1-8e616ff90b00" (UID: "dfe944db-5201-4af3-8bb1-8e616ff90b00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:03:40.450846 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.450821 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe944db-5201-4af3-8bb1-8e616ff90b00-kube-api-access-h6jg7" pod "dfe944db-5201-4af3-8bb1-8e616ff90b00" (UID: "dfe944db-5201-4af3-8bb1-8e616ff90b00"). InnerVolumeSpecName "kube-api-access-h6jg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:03:40.550487 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.550441 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3aece38-123d-42a2-a192-298b1d6b1579-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3aece38-123d-42a2-a192-298b1d6b1579-plugins-config-volume\") pod \"c3aece38-123d-42a2-a192-298b1d6b1579\" (UID: \"c3aece38-123d-42a2-a192-298b1d6b1579\") " Oct 30 19:03:40.550704 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.550678 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3aece38-123d-42a2-a192-298b1d6b1579-kube-api-access-9sjt7\" (UniqueName: \"kubernetes.io/projected/c3aece38-123d-42a2-a192-298b1d6b1579-kube-api-access-9sjt7\") pod \"c3aece38-123d-42a2-a192-298b1d6b1579\" (UID: \"c3aece38-123d-42a2-a192-298b1d6b1579\") " Oct 30 19:03:40.550935 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.550902 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3aece38-123d-42a2-a192-298b1d6b1579-plugins-config-volume" pod "c3aece38-123d-42a2-a192-298b1d6b1579" (UID: "c3aece38-123d-42a2-a192-298b1d6b1579"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:03:40.551235 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.551214 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfe944db-5201-4af3-8bb1-8e616ff90b00-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:40.551306 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.551240 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6jg7\" (UniqueName: \"kubernetes.io/projected/dfe944db-5201-4af3-8bb1-8e616ff90b00-kube-api-access-h6jg7\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:40.551306 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.551255 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3aece38-123d-42a2-a192-298b1d6b1579-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:40.553098 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.553065 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3aece38-123d-42a2-a192-298b1d6b1579-kube-api-access-9sjt7" pod "c3aece38-123d-42a2-a192-298b1d6b1579" (UID: "c3aece38-123d-42a2-a192-298b1d6b1579"). InnerVolumeSpecName "kube-api-access-9sjt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:03:40.652853 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.652777 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sjt7\" (UniqueName: \"kubernetes.io/projected/c3aece38-123d-42a2-a192-298b1d6b1579-kube-api-access-9sjt7\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:03:40.808829 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.808794 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc"] Oct 30 19:03:40.813075 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:40.813046 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4ebcd2-d8dd6b4cb-cpfwc"] Oct 30 19:03:41.398520 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:41.398434 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm" Oct 30 19:03:41.398520 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:41.398451 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm" event={"ID":"c3aece38-123d-42a2-a192-298b1d6b1579","Type":"ContainerDied","Data":"81554567542e847abfbae2f9dae0a60ab4a1fa95b5dcc2500b2c35dd00ed9f0f"} Oct 30 19:03:41.398520 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:41.398518 4531 scope.go:122] "RemoveContainer" containerID="ef5eb488805d44a6b4fc4e568334eebf696f1319328db5e302f929bb39fc9d8e" Oct 30 19:03:41.564044 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:41.564004 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm"] Oct 30 19:03:41.613490 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:41.613454 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4ebcd2-cf9f75c75-l42sm"] Oct 30 19:03:42.098790 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:42.098756 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3aece38-123d-42a2-a192-298b1d6b1579" path="/var/lib/kubelet/pods/c3aece38-123d-42a2-a192-298b1d6b1579/volumes" Oct 30 19:03:42.099191 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:03:42.099177 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfe944db-5201-4af3-8bb1-8e616ff90b00" path="/var/lib/kubelet/pods/dfe944db-5201-4af3-8bb1-8e616ff90b00/volumes" Oct 30 19:03:56.399832 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:03:56.399793 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:04:01.330005 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:01.329972 4531 scope.go:122] "RemoveContainer" containerID="821ddcf1e4e0ecf716153acce6b0d1a79c60c645a7e5b2872997b498ba5ec03e" Oct 30 19:04:08.409349 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:08.409305 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-132af1-zsd7z"] Oct 30 19:04:08.458027 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:08.457991 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-132af1-zsd7z"] Oct 30 19:04:10.099408 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:10.099368 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b2ed0c4-859a-41eb-9391-8720e3b58365" path="/var/lib/kubelet/pods/2b2ed0c4-859a-41eb-9391-8720e3b58365/volumes" Oct 30 19:04:16.546267 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:16.546232 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm"] Oct 30 19:04:16.546778 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:16.546500 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" podUID="52581cb3-251f-4450-b981-1b7fa19d9367" containerName="istio-proxy" containerID="cri-o://83f1bbe34104c814717834a5b147f817d54fa785254909f4e8652160c93f437f" gracePeriod=30 Oct 30 19:04:21.767285 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:21.767244 4531 generic.go:356] "Generic (PLEG): container finished" podID="52581cb3-251f-4450-b981-1b7fa19d9367" containerID="83f1bbe34104c814717834a5b147f817d54fa785254909f4e8652160c93f437f" exitCode=0 Oct 30 19:04:21.767750 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:21.767320 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" event={"ID":"52581cb3-251f-4450-b981-1b7fa19d9367","Type":"ContainerDied","Data":"83f1bbe34104c814717834a5b147f817d54fa785254909f4e8652160c93f437f"} Oct 30 19:04:21.767750 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:21.767442 4531 scope.go:122] "RemoveContainer" containerID="28de017c8644ce0e679c153d9a2e123e3d8940c1bfb92cec5b413eae59930edc" Oct 30 19:04:21.822837 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:21.822770 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 19:04:22.004067 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.004038 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-istio-envoy\") pod \"52581cb3-251f-4450-b981-1b7fa19d9367\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " Oct 30 19:04:22.004278 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.004123 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-istio-data\") pod \"52581cb3-251f-4450-b981-1b7fa19d9367\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " Oct 30 19:04:22.004278 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.004219 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-workload-certs\") pod \"52581cb3-251f-4450-b981-1b7fa19d9367\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " Oct 30 19:04:22.004403 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.004277 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52581cb3-251f-4450-b981-1b7fa19d9367-kube-api-access-8pkbw\" (UniqueName: \"kubernetes.io/projected/52581cb3-251f-4450-b981-1b7fa19d9367-kube-api-access-8pkbw\") pod \"52581cb3-251f-4450-b981-1b7fa19d9367\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " Oct 30 19:04:22.004403 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.004307 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52581cb3-251f-4450-b981-1b7fa19d9367-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52581cb3-251f-4450-b981-1b7fa19d9367-istiod-ca-cert\") pod \"52581cb3-251f-4450-b981-1b7fa19d9367\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " Oct 30 19:04:22.004403 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.004366 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-credential-socket\") pod \"52581cb3-251f-4450-b981-1b7fa19d9367\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " Oct 30 19:04:22.004555 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.004418 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52581cb3-251f-4450-b981-1b7fa19d9367-istio-token\" (UniqueName: \"kubernetes.io/projected/52581cb3-251f-4450-b981-1b7fa19d9367-istio-token\") pod \"52581cb3-251f-4450-b981-1b7fa19d9367\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " Oct 30 19:04:22.004555 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.004500 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-workload-socket\") pod \"52581cb3-251f-4450-b981-1b7fa19d9367\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " Oct 30 19:04:22.004555 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.004532 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52581cb3-251f-4450-b981-1b7fa19d9367-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52581cb3-251f-4450-b981-1b7fa19d9367-istio-podinfo\") pod \"52581cb3-251f-4450-b981-1b7fa19d9367\" (UID: \"52581cb3-251f-4450-b981-1b7fa19d9367\") " Oct 30 19:04:22.004713 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.004535 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-istio-data" pod "52581cb3-251f-4450-b981-1b7fa19d9367" (UID: "52581cb3-251f-4450-b981-1b7fa19d9367"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:22.004713 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.004606 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-workload-certs" pod "52581cb3-251f-4450-b981-1b7fa19d9367" (UID: "52581cb3-251f-4450-b981-1b7fa19d9367"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:22.005601 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.004825 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-credential-socket" pod "52581cb3-251f-4450-b981-1b7fa19d9367" (UID: "52581cb3-251f-4450-b981-1b7fa19d9367"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:22.005601 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.005314 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:04:22.005601 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.005338 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:04:22.005601 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.005352 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:04:22.005601 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.005480 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52581cb3-251f-4450-b981-1b7fa19d9367-istiod-ca-cert" pod "52581cb3-251f-4450-b981-1b7fa19d9367" (UID: "52581cb3-251f-4450-b981-1b7fa19d9367"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:04:22.005601 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.005574 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-workload-socket" pod "52581cb3-251f-4450-b981-1b7fa19d9367" (UID: "52581cb3-251f-4450-b981-1b7fa19d9367"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:22.007202 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.007160 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52581cb3-251f-4450-b981-1b7fa19d9367-istio-podinfo" pod "52581cb3-251f-4450-b981-1b7fa19d9367" (UID: "52581cb3-251f-4450-b981-1b7fa19d9367"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:04:22.007443 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.007419 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52581cb3-251f-4450-b981-1b7fa19d9367-kube-api-access-8pkbw" pod "52581cb3-251f-4450-b981-1b7fa19d9367" (UID: "52581cb3-251f-4450-b981-1b7fa19d9367"). InnerVolumeSpecName "kube-api-access-8pkbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:22.007698 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.007604 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52581cb3-251f-4450-b981-1b7fa19d9367-istio-token" pod "52581cb3-251f-4450-b981-1b7fa19d9367" (UID: "52581cb3-251f-4450-b981-1b7fa19d9367"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:22.007698 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.007678 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-istio-envoy" pod "52581cb3-251f-4450-b981-1b7fa19d9367" (UID: "52581cb3-251f-4450-b981-1b7fa19d9367"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:22.106806 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.106719 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pkbw\" (UniqueName: \"kubernetes.io/projected/52581cb3-251f-4450-b981-1b7fa19d9367-kube-api-access-8pkbw\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:04:22.106806 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.106755 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52581cb3-251f-4450-b981-1b7fa19d9367-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:04:22.106806 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.106766 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52581cb3-251f-4450-b981-1b7fa19d9367-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:04:22.106806 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.106775 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:04:22.106806 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.106784 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52581cb3-251f-4450-b981-1b7fa19d9367-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:04:22.106806 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.106793 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52581cb3-251f-4450-b981-1b7fa19d9367-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:04:22.775427 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.775398 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" Oct 30 19:04:22.775935 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.775429 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm" event={"ID":"52581cb3-251f-4450-b981-1b7fa19d9367","Type":"ContainerDied","Data":"bd19d4a9434a5cc87beb6ff3b0ff99fe8b2b5ffe1f8cbfba993c5d91aae3e4bb"} Oct 30 19:04:22.775935 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.775495 4531 scope.go:122] "RemoveContainer" containerID="83f1bbe34104c814717834a5b147f817d54fa785254909f4e8652160c93f437f" Oct 30 19:04:22.992252 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:22.992212 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm"] Oct 30 19:04:23.041736 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:23.041653 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-132af1-openshift-default-5754d6cd99-nx8sm"] Oct 30 19:04:24.099939 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:24.099899 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52581cb3-251f-4450-b981-1b7fa19d9367" path="/var/lib/kubelet/pods/52581cb3-251f-4450-b981-1b7fa19d9367/volumes" Oct 30 19:04:26.399162 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:04:26.399103 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:04:52.770737 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:52.770653 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q"] Oct 30 19:04:52.820041 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:52.820009 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ebcd2-wz99q"] Oct 30 19:04:54.098929 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:04:54.098891 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58ca32e1-9bcb-4600-b2ad-17e3656c4db6" path="/var/lib/kubelet/pods/58ca32e1-9bcb-4600-b2ad-17e3656c4db6/volumes" Oct 30 19:04:56.398098 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:04:56.398060 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:05:01.363830 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:01.363791 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd"] Oct 30 19:05:01.364459 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:01.364039 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" podUID="b7ea737e-8a33-495c-959e-a037aed04436" containerName="istio-proxy" containerID="cri-o://367a04ed9f6a810e261834b3910bc109784682a4d2eb0b3b633bac5c377088ed" gracePeriod=30 Oct 30 19:05:01.368267 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:01.368228 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg"] Oct 30 19:05:01.368563 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:01.368533 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" podUID="0005aee3-3a38-431e-b86c-d1c40bab8483" containerName="istio-proxy" containerID="cri-o://6740d54ef5d500dad169fdbb4059b728df2209f27c3952d009654f7a9fdac402" gracePeriod=30 Oct 30 19:05:01.385330 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:01.385285 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg"] Oct 30 19:05:01.385613 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:01.385581 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" podUID="2ed7421d-881e-4a42-a128-e4852abe4cfd" containerName="istio-proxy" containerID="cri-o://43a351f28a72b65aa3439ddbb0d834d61a721caf1011386af9ace1ed9f907d6e" gracePeriod=30 Oct 30 19:05:01.387446 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:01.387406 4531 scope.go:122] "RemoveContainer" containerID="46df501a3552f694cacce424ddbd4aef1f77f47bdfad9df5e5b417d87a559e86" Oct 30 19:05:01.399181 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:01.399109 4531 scope.go:122] "RemoveContainer" containerID="2e3eac8993e944b0f77e00199f578421f96a1acf54c5dc25c401384a292b8551" Oct 30 19:05:01.853892 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:01.853851 4531 patch_prober.go:29] interesting pod/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:05:01.854154 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:01.853923 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" podUID="0005aee3-3a38-431e-b86c-d1c40bab8483" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:05:01.971081 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:01.971029 4531 patch_prober.go:29] interesting pod/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:05:01.971359 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:01.971098 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" podUID="2ed7421d-881e-4a42-a128-e4852abe4cfd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:05:02.580613 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:02.580567 4531 patch_prober.go:29] interesting pod/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:05:02.581083 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:02.580642 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" podUID="b7ea737e-8a33-495c-959e-a037aed04436" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:05:06.679076 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.679036 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:05:06.704937 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.704907 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:05:06.743691 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.743654 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:05:06.744728 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.744704 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-istio-data\") pod \"b7ea737e-8a33-495c-959e-a037aed04436\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " Oct 30 19:05:06.744809 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.744756 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7ea737e-8a33-495c-959e-a037aed04436-kube-api-access-678fl\" (UniqueName: \"kubernetes.io/projected/b7ea737e-8a33-495c-959e-a037aed04436-kube-api-access-678fl\") pod \"b7ea737e-8a33-495c-959e-a037aed04436\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " Oct 30 19:05:06.744809 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.744799 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-istio-envoy\") pod \"b7ea737e-8a33-495c-959e-a037aed04436\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " Oct 30 19:05:06.744895 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.744838 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-workload-certs\") pod \"b7ea737e-8a33-495c-959e-a037aed04436\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " Oct 30 19:05:06.744949 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.744899 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7ea737e-8a33-495c-959e-a037aed04436-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7ea737e-8a33-495c-959e-a037aed04436-istio-podinfo\") pod \"b7ea737e-8a33-495c-959e-a037aed04436\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " Oct 30 19:05:06.745004 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.744962 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-credential-socket\") pod \"b7ea737e-8a33-495c-959e-a037aed04436\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " Oct 30 19:05:06.745055 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.745042 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7ea737e-8a33-495c-959e-a037aed04436-istio-token\" (UniqueName: \"kubernetes.io/projected/b7ea737e-8a33-495c-959e-a037aed04436-istio-token\") pod \"b7ea737e-8a33-495c-959e-a037aed04436\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " Oct 30 19:05:06.745108 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.745062 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-istio-data" pod "b7ea737e-8a33-495c-959e-a037aed04436" (UID: "b7ea737e-8a33-495c-959e-a037aed04436"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:06.745207 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.745176 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7ea737e-8a33-495c-959e-a037aed04436-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7ea737e-8a33-495c-959e-a037aed04436-istiod-ca-cert\") pod \"b7ea737e-8a33-495c-959e-a037aed04436\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " Oct 30 19:05:06.745293 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.745281 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-workload-socket\") pod \"b7ea737e-8a33-495c-959e-a037aed04436\" (UID: \"b7ea737e-8a33-495c-959e-a037aed04436\") " Oct 30 19:05:06.745353 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.745315 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-credential-socket" pod "b7ea737e-8a33-495c-959e-a037aed04436" (UID: "b7ea737e-8a33-495c-959e-a037aed04436"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:06.745439 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.745416 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-workload-certs" pod "b7ea737e-8a33-495c-959e-a037aed04436" (UID: "b7ea737e-8a33-495c-959e-a037aed04436"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:06.745548 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.745532 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-workload-socket" pod "b7ea737e-8a33-495c-959e-a037aed04436" (UID: "b7ea737e-8a33-495c-959e-a037aed04436"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:06.745636 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.745599 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7ea737e-8a33-495c-959e-a037aed04436-istiod-ca-cert" pod "b7ea737e-8a33-495c-959e-a037aed04436" (UID: "b7ea737e-8a33-495c-959e-a037aed04436"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:06.745883 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.745865 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:06.745961 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.745889 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:06.745961 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.745903 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:06.745961 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.745917 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:06.745961 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.745931 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7ea737e-8a33-495c-959e-a037aed04436-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:06.747431 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.747392 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7ea737e-8a33-495c-959e-a037aed04436-kube-api-access-678fl" pod "b7ea737e-8a33-495c-959e-a037aed04436" (UID: "b7ea737e-8a33-495c-959e-a037aed04436"). InnerVolumeSpecName "kube-api-access-678fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:06.747838 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.747804 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7ea737e-8a33-495c-959e-a037aed04436-istio-podinfo" pod "b7ea737e-8a33-495c-959e-a037aed04436" (UID: "b7ea737e-8a33-495c-959e-a037aed04436"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:05:06.747838 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.747818 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7ea737e-8a33-495c-959e-a037aed04436-istio-token" pod "b7ea737e-8a33-495c-959e-a037aed04436" (UID: "b7ea737e-8a33-495c-959e-a037aed04436"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:06.747964 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.747874 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-istio-envoy" pod "b7ea737e-8a33-495c-959e-a037aed04436" (UID: "b7ea737e-8a33-495c-959e-a037aed04436"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:06.846684 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.846642 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-envoy\") pod \"0005aee3-3a38-431e-b86c-d1c40bab8483\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " Oct 30 19:05:06.846887 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.846715 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-credential-socket\") pod \"2ed7421d-881e-4a42-a128-e4852abe4cfd\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " Oct 30 19:05:06.846887 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.846738 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-data\") pod \"0005aee3-3a38-431e-b86c-d1c40bab8483\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " Oct 30 19:05:06.846887 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.846763 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-workload-certs\") pod \"2ed7421d-881e-4a42-a128-e4852abe4cfd\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " Oct 30 19:05:06.846887 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.846798 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-podinfo\") pod \"2ed7421d-881e-4a42-a128-e4852abe4cfd\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " Oct 30 19:05:06.846887 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.846830 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-token\" (UniqueName: \"kubernetes.io/projected/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-token\") pod \"0005aee3-3a38-431e-b86c-d1c40bab8483\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " Oct 30 19:05:06.847212 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.846909 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ed7421d-881e-4a42-a128-e4852abe4cfd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed7421d-881e-4a42-a128-e4852abe4cfd-istiod-ca-cert\") pod \"2ed7421d-881e-4a42-a128-e4852abe4cfd\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " Oct 30 19:05:06.847212 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.846947 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ed7421d-881e-4a42-a128-e4852abe4cfd-kube-api-access-gngv4\" (UniqueName: \"kubernetes.io/projected/2ed7421d-881e-4a42-a128-e4852abe4cfd-kube-api-access-gngv4\") pod \"2ed7421d-881e-4a42-a128-e4852abe4cfd\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " Oct 30 19:05:06.847212 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.847006 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-credential-socket\") pod \"0005aee3-3a38-431e-b86c-d1c40bab8483\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " Oct 30 19:05:06.847212 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.847050 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-workload-certs\") pod \"0005aee3-3a38-431e-b86c-d1c40bab8483\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " Oct 30 19:05:06.847212 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.847062 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-data" pod "0005aee3-3a38-431e-b86c-d1c40bab8483" (UID: "0005aee3-3a38-431e-b86c-d1c40bab8483"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:06.847212 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.847077 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-workload-certs" pod "2ed7421d-881e-4a42-a128-e4852abe4cfd" (UID: "2ed7421d-881e-4a42-a128-e4852abe4cfd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:06.847212 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.847057 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-credential-socket" pod "2ed7421d-881e-4a42-a128-e4852abe4cfd" (UID: "2ed7421d-881e-4a42-a128-e4852abe4cfd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:06.847212 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.847086 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-data\") pod \"2ed7421d-881e-4a42-a128-e4852abe4cfd\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " Oct 30 19:05:06.847212 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.847210 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0005aee3-3a38-431e-b86c-d1c40bab8483-kube-api-access-sl62d\" (UniqueName: \"kubernetes.io/projected/0005aee3-3a38-431e-b86c-d1c40bab8483-kube-api-access-sl62d\") pod \"0005aee3-3a38-431e-b86c-d1c40bab8483\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " Oct 30 19:05:06.847674 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.847294 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-token\" (UniqueName: \"kubernetes.io/projected/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-token\") pod \"2ed7421d-881e-4a42-a128-e4852abe4cfd\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " Oct 30 19:05:06.847674 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.847341 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-credential-socket" pod "0005aee3-3a38-431e-b86c-d1c40bab8483" (UID: "0005aee3-3a38-431e-b86c-d1c40bab8483"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:06.847674 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.847353 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-data" pod "2ed7421d-881e-4a42-a128-e4852abe4cfd" (UID: "2ed7421d-881e-4a42-a128-e4852abe4cfd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:06.847674 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.847364 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-workload-socket\") pod \"0005aee3-3a38-431e-b86c-d1c40bab8483\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " Oct 30 19:05:06.847674 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.847398 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0005aee3-3a38-431e-b86c-d1c40bab8483-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0005aee3-3a38-431e-b86c-d1c40bab8483-istiod-ca-cert\") pod \"0005aee3-3a38-431e-b86c-d1c40bab8483\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " Oct 30 19:05:06.847674 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.847425 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-podinfo\") pod \"0005aee3-3a38-431e-b86c-d1c40bab8483\" (UID: \"0005aee3-3a38-431e-b86c-d1c40bab8483\") " Oct 30 19:05:06.847674 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.847525 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-envoy\") pod \"2ed7421d-881e-4a42-a128-e4852abe4cfd\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " Oct 30 19:05:06.847674 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.847578 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-workload-socket\") pod \"2ed7421d-881e-4a42-a128-e4852abe4cfd\" (UID: \"2ed7421d-881e-4a42-a128-e4852abe4cfd\") " Oct 30 19:05:06.847674 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.847663 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-workload-certs" pod "0005aee3-3a38-431e-b86c-d1c40bab8483" (UID: "0005aee3-3a38-431e-b86c-d1c40bab8483"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:06.848176 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.847710 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ed7421d-881e-4a42-a128-e4852abe4cfd-istiod-ca-cert" pod "2ed7421d-881e-4a42-a128-e4852abe4cfd" (UID: "2ed7421d-881e-4a42-a128-e4852abe4cfd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:06.848176 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.848030 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0005aee3-3a38-431e-b86c-d1c40bab8483-istiod-ca-cert" pod "0005aee3-3a38-431e-b86c-d1c40bab8483" (UID: "0005aee3-3a38-431e-b86c-d1c40bab8483"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:06.848632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.848425 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:06.848632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.848454 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:06.848632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.848468 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:06.848632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.848483 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed7421d-881e-4a42-a128-e4852abe4cfd-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:06.848632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.848497 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-678fl\" (UniqueName: \"kubernetes.io/projected/b7ea737e-8a33-495c-959e-a037aed04436-kube-api-access-678fl\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:06.848632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.848512 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:06.848632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.848525 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7ea737e-8a33-495c-959e-a037aed04436-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:06.848632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.848523 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-workload-socket" pod "0005aee3-3a38-431e-b86c-d1c40bab8483" (UID: "0005aee3-3a38-431e-b86c-d1c40bab8483"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:06.848632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.848541 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:06.848632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.848557 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:06.848632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.848570 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7ea737e-8a33-495c-959e-a037aed04436-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:06.848632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.848584 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0005aee3-3a38-431e-b86c-d1c40bab8483-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:06.848632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.848598 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7ea737e-8a33-495c-959e-a037aed04436-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:06.849512 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.848695 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-workload-socket" pod "2ed7421d-881e-4a42-a128-e4852abe4cfd" (UID: "2ed7421d-881e-4a42-a128-e4852abe4cfd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:06.850019 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.849949 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed7421d-881e-4a42-a128-e4852abe4cfd-kube-api-access-gngv4" pod "2ed7421d-881e-4a42-a128-e4852abe4cfd" (UID: "2ed7421d-881e-4a42-a128-e4852abe4cfd"). InnerVolumeSpecName "kube-api-access-gngv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:06.850245 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.850215 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-envoy" pod "0005aee3-3a38-431e-b86c-d1c40bab8483" (UID: "0005aee3-3a38-431e-b86c-d1c40bab8483"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:06.850359 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.850267 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-podinfo" pod "0005aee3-3a38-431e-b86c-d1c40bab8483" (UID: "0005aee3-3a38-431e-b86c-d1c40bab8483"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:05:06.850426 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.850396 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0005aee3-3a38-431e-b86c-d1c40bab8483-kube-api-access-sl62d" pod "0005aee3-3a38-431e-b86c-d1c40bab8483" (UID: "0005aee3-3a38-431e-b86c-d1c40bab8483"). InnerVolumeSpecName "kube-api-access-sl62d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:06.850523 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.850431 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-podinfo" pod "2ed7421d-881e-4a42-a128-e4852abe4cfd" (UID: "2ed7421d-881e-4a42-a128-e4852abe4cfd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:05:06.850688 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.850664 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-token" pod "2ed7421d-881e-4a42-a128-e4852abe4cfd" (UID: "2ed7421d-881e-4a42-a128-e4852abe4cfd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:06.851347 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.851319 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-token" pod "0005aee3-3a38-431e-b86c-d1c40bab8483" (UID: "0005aee3-3a38-431e-b86c-d1c40bab8483"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:06.851883 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.851865 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-envoy" pod "2ed7421d-881e-4a42-a128-e4852abe4cfd" (UID: "2ed7421d-881e-4a42-a128-e4852abe4cfd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:06.949472 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.949355 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gngv4\" (UniqueName: \"kubernetes.io/projected/2ed7421d-881e-4a42-a128-e4852abe4cfd-kube-api-access-gngv4\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:06.949472 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.949397 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl62d\" (UniqueName: \"kubernetes.io/projected/0005aee3-3a38-431e-b86c-d1c40bab8483-kube-api-access-sl62d\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:06.949472 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.949409 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:06.949472 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.949420 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:06.949472 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.949428 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:06.949472 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.949439 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:06.949472 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.949447 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed7421d-881e-4a42-a128-e4852abe4cfd-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:06.949472 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.949455 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:06.949472 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.949463 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed7421d-881e-4a42-a128-e4852abe4cfd-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:06.949472 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:06.949473 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0005aee3-3a38-431e-b86c-d1c40bab8483-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:07.125968 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.125923 4531 generic.go:356] "Generic (PLEG): container finished" podID="2ed7421d-881e-4a42-a128-e4852abe4cfd" containerID="43a351f28a72b65aa3439ddbb0d834d61a721caf1011386af9ace1ed9f907d6e" exitCode=0 Oct 30 19:05:07.126412 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.126020 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" Oct 30 19:05:07.133199 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.126020 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" event={"ID":"2ed7421d-881e-4a42-a128-e4852abe4cfd","Type":"ContainerDied","Data":"43a351f28a72b65aa3439ddbb0d834d61a721caf1011386af9ace1ed9f907d6e"} Oct 30 19:05:07.133398 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.133308 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg" event={"ID":"2ed7421d-881e-4a42-a128-e4852abe4cfd","Type":"ContainerDied","Data":"520b17a557812536591d9dfca2c0f664b6a5671a47396a07e9edcf4032fff012"} Oct 30 19:05:07.133398 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.133375 4531 scope.go:122] "RemoveContainer" containerID="43a351f28a72b65aa3439ddbb0d834d61a721caf1011386af9ace1ed9f907d6e" Oct 30 19:05:07.136412 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.136379 4531 generic.go:356] "Generic (PLEG): container finished" podID="0005aee3-3a38-431e-b86c-d1c40bab8483" containerID="6740d54ef5d500dad169fdbb4059b728df2209f27c3952d009654f7a9fdac402" exitCode=0 Oct 30 19:05:07.136576 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.136564 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" Oct 30 19:05:07.139221 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.139184 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" event={"ID":"0005aee3-3a38-431e-b86c-d1c40bab8483","Type":"ContainerDied","Data":"6740d54ef5d500dad169fdbb4059b728df2209f27c3952d009654f7a9fdac402"} Oct 30 19:05:07.139387 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.139260 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg" event={"ID":"0005aee3-3a38-431e-b86c-d1c40bab8483","Type":"ContainerDied","Data":"30b8ac56c0e960fdac4578828c78d2903237c9c9cdbe58b5bee3d672afb0f56b"} Oct 30 19:05:07.141505 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.141448 4531 generic.go:356] "Generic (PLEG): container finished" podID="b7ea737e-8a33-495c-959e-a037aed04436" containerID="367a04ed9f6a810e261834b3910bc109784682a4d2eb0b3b633bac5c377088ed" exitCode=0 Oct 30 19:05:07.141673 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.141536 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" Oct 30 19:05:07.141673 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.141549 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" event={"ID":"b7ea737e-8a33-495c-959e-a037aed04436","Type":"ContainerDied","Data":"367a04ed9f6a810e261834b3910bc109784682a4d2eb0b3b633bac5c377088ed"} Oct 30 19:05:07.141673 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.141600 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd" event={"ID":"b7ea737e-8a33-495c-959e-a037aed04436","Type":"ContainerDied","Data":"51d11681020a26022d374cbca8dd47ff2fe75a0a338cdcbba69140175f544ac8"} Oct 30 19:05:07.150420 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.150386 4531 scope.go:122] "RemoveContainer" containerID="43a351f28a72b65aa3439ddbb0d834d61a721caf1011386af9ace1ed9f907d6e" Oct 30 19:05:07.151392 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:05:07.151361 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43a351f28a72b65aa3439ddbb0d834d61a721caf1011386af9ace1ed9f907d6e\": container with ID starting with 43a351f28a72b65aa3439ddbb0d834d61a721caf1011386af9ace1ed9f907d6e not found: ID does not exist" containerID="43a351f28a72b65aa3439ddbb0d834d61a721caf1011386af9ace1ed9f907d6e" Oct 30 19:05:07.151523 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.151407 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43a351f28a72b65aa3439ddbb0d834d61a721caf1011386af9ace1ed9f907d6e"} err="failed to get container status \"43a351f28a72b65aa3439ddbb0d834d61a721caf1011386af9ace1ed9f907d6e\": rpc error: code = NotFound desc = could not find container \"43a351f28a72b65aa3439ddbb0d834d61a721caf1011386af9ace1ed9f907d6e\": container with ID starting with 43a351f28a72b65aa3439ddbb0d834d61a721caf1011386af9ace1ed9f907d6e not found: ID does not exist" Oct 30 19:05:07.151523 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.151435 4531 scope.go:122] "RemoveContainer" containerID="6740d54ef5d500dad169fdbb4059b728df2209f27c3952d009654f7a9fdac402" Oct 30 19:05:07.164107 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.164084 4531 scope.go:122] "RemoveContainer" containerID="6740d54ef5d500dad169fdbb4059b728df2209f27c3952d009654f7a9fdac402" Oct 30 19:05:07.164522 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:05:07.164495 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6740d54ef5d500dad169fdbb4059b728df2209f27c3952d009654f7a9fdac402\": container with ID starting with 6740d54ef5d500dad169fdbb4059b728df2209f27c3952d009654f7a9fdac402 not found: ID does not exist" containerID="6740d54ef5d500dad169fdbb4059b728df2209f27c3952d009654f7a9fdac402" Oct 30 19:05:07.164629 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.164535 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6740d54ef5d500dad169fdbb4059b728df2209f27c3952d009654f7a9fdac402"} err="failed to get container status \"6740d54ef5d500dad169fdbb4059b728df2209f27c3952d009654f7a9fdac402\": rpc error: code = NotFound desc = could not find container \"6740d54ef5d500dad169fdbb4059b728df2209f27c3952d009654f7a9fdac402\": container with ID starting with 6740d54ef5d500dad169fdbb4059b728df2209f27c3952d009654f7a9fdac402 not found: ID does not exist" Oct 30 19:05:07.164629 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.164562 4531 scope.go:122] "RemoveContainer" containerID="367a04ed9f6a810e261834b3910bc109784682a4d2eb0b3b633bac5c377088ed" Oct 30 19:05:07.175230 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.175205 4531 scope.go:122] "RemoveContainer" containerID="367a04ed9f6a810e261834b3910bc109784682a4d2eb0b3b633bac5c377088ed" Oct 30 19:05:07.175802 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:05:07.175773 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"367a04ed9f6a810e261834b3910bc109784682a4d2eb0b3b633bac5c377088ed\": container with ID starting with 367a04ed9f6a810e261834b3910bc109784682a4d2eb0b3b633bac5c377088ed not found: ID does not exist" containerID="367a04ed9f6a810e261834b3910bc109784682a4d2eb0b3b633bac5c377088ed" Oct 30 19:05:07.175890 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.175813 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"367a04ed9f6a810e261834b3910bc109784682a4d2eb0b3b633bac5c377088ed"} err="failed to get container status \"367a04ed9f6a810e261834b3910bc109784682a4d2eb0b3b633bac5c377088ed\": rpc error: code = NotFound desc = could not find container \"367a04ed9f6a810e261834b3910bc109784682a4d2eb0b3b633bac5c377088ed\": container with ID starting with 367a04ed9f6a810e261834b3910bc109784682a4d2eb0b3b633bac5c377088ed not found: ID does not exist" Oct 30 19:05:07.452146 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.452093 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg"] Oct 30 19:05:07.466223 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.466174 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-mrgtg"] Oct 30 19:05:07.592602 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.592193 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg"] Oct 30 19:05:07.624984 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.624936 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-92ffg"] Oct 30 19:05:07.691990 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.691892 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd"] Oct 30 19:05:07.697491 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:07.697443 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4ebcd2-openshift-default-59d766c94c-t7xqd"] Oct 30 19:05:08.099459 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:08.099419 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0005aee3-3a38-431e-b86c-d1c40bab8483" path="/var/lib/kubelet/pods/0005aee3-3a38-431e-b86c-d1c40bab8483/volumes" Oct 30 19:05:08.099955 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:08.099938 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ed7421d-881e-4a42-a128-e4852abe4cfd" path="/var/lib/kubelet/pods/2ed7421d-881e-4a42-a128-e4852abe4cfd/volumes" Oct 30 19:05:08.100460 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:08.100443 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7ea737e-8a33-495c-959e-a037aed04436" path="/var/lib/kubelet/pods/b7ea737e-8a33-495c-959e-a037aed04436/volumes" Oct 30 19:05:13.400053 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:13.400015 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779"] Oct 30 19:05:13.400499 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:13.400257 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" podUID="26466126-8cc0-4784-9c40-a2d16bc3256d" containerName="istio-proxy" containerID="cri-o://6f425a94e5c2ae0ef2a8eff0adf72cc34402be61053ee1b9d85e4c3908d95676" gracePeriod=30 Oct 30 19:05:16.797110 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:16.797065 4531 patch_prober.go:29] interesting pod/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:05:16.797566 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:16.797185 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" podUID="26466126-8cc0-4784-9c40-a2d16bc3256d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:05:17.395155 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:17.393688 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q"] Oct 30 19:05:17.395155 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:17.393970 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" podUID="4adfad0a-5914-442f-afe1-c7f9c2f70849" containerName="istio-proxy" containerID="cri-o://8b5a37b986be2c06b49f1d324cea81721b09431bc76379c7d7d3cee807f7eb19" gracePeriod=30 Oct 30 19:05:18.272431 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.272390 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh"] Oct 30 19:05:18.273371 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.273300 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" podUID="8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6" containerName="istio-proxy" containerID="cri-o://bcd8a7d479e675dbf063838c670abb5163e72cd4035c52154867fede86c59f4d" gracePeriod=30 Oct 30 19:05:18.275417 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.275382 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265"] Oct 30 19:05:18.275637 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.275587 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" podUID="56bbae65-edbd-48ad-b1f6-7c5da8d2d53d" containerName="istio-proxy" containerID="cri-o://305ec0fdca540995d5b873ff12e609e82d95ccb2bf3d18d015d7a60c2e8d95ef" gracePeriod=30 Oct 30 19:05:18.687154 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.687079 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 19:05:18.705052 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.705015 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-podinfo\") pod \"26466126-8cc0-4784-9c40-a2d16bc3256d\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " Oct 30 19:05:18.705289 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.705069 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-envoy\") pod \"26466126-8cc0-4784-9c40-a2d16bc3256d\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " Oct 30 19:05:18.705289 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.705091 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-token\" (UniqueName: \"kubernetes.io/projected/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-token\") pod \"26466126-8cc0-4784-9c40-a2d16bc3256d\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " Oct 30 19:05:18.705289 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.705174 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26466126-8cc0-4784-9c40-a2d16bc3256d-kube-api-access-xrrct\" (UniqueName: \"kubernetes.io/projected/26466126-8cc0-4784-9c40-a2d16bc3256d-kube-api-access-xrrct\") pod \"26466126-8cc0-4784-9c40-a2d16bc3256d\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " Oct 30 19:05:18.705289 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.705221 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-credential-socket\") pod \"26466126-8cc0-4784-9c40-a2d16bc3256d\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " Oct 30 19:05:18.705289 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.705269 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-data\") pod \"26466126-8cc0-4784-9c40-a2d16bc3256d\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " Oct 30 19:05:18.705586 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.705317 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-workload-certs\") pod \"26466126-8cc0-4784-9c40-a2d16bc3256d\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " Oct 30 19:05:18.705586 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.705375 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26466126-8cc0-4784-9c40-a2d16bc3256d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26466126-8cc0-4784-9c40-a2d16bc3256d-istiod-ca-cert\") pod \"26466126-8cc0-4784-9c40-a2d16bc3256d\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " Oct 30 19:05:18.705586 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.705418 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-workload-socket\") pod \"26466126-8cc0-4784-9c40-a2d16bc3256d\" (UID: \"26466126-8cc0-4784-9c40-a2d16bc3256d\") " Oct 30 19:05:18.705732 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.705607 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-data" pod "26466126-8cc0-4784-9c40-a2d16bc3256d" (UID: "26466126-8cc0-4784-9c40-a2d16bc3256d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:18.705732 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.705622 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-workload-certs" pod "26466126-8cc0-4784-9c40-a2d16bc3256d" (UID: "26466126-8cc0-4784-9c40-a2d16bc3256d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:18.705839 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.705816 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-workload-socket" pod "26466126-8cc0-4784-9c40-a2d16bc3256d" (UID: "26466126-8cc0-4784-9c40-a2d16bc3256d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:18.706115 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.705915 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-credential-socket" pod "26466126-8cc0-4784-9c40-a2d16bc3256d" (UID: "26466126-8cc0-4784-9c40-a2d16bc3256d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:18.706115 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.706063 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:18.706115 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.706081 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:18.706115 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.706097 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:18.706445 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.706192 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26466126-8cc0-4784-9c40-a2d16bc3256d-istiod-ca-cert" pod "26466126-8cc0-4784-9c40-a2d16bc3256d" (UID: "26466126-8cc0-4784-9c40-a2d16bc3256d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:18.707723 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.707688 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-podinfo" pod "26466126-8cc0-4784-9c40-a2d16bc3256d" (UID: "26466126-8cc0-4784-9c40-a2d16bc3256d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:05:18.707845 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.707796 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26466126-8cc0-4784-9c40-a2d16bc3256d-kube-api-access-xrrct" pod "26466126-8cc0-4784-9c40-a2d16bc3256d" (UID: "26466126-8cc0-4784-9c40-a2d16bc3256d"). InnerVolumeSpecName "kube-api-access-xrrct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:18.708083 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.708059 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-envoy" pod "26466126-8cc0-4784-9c40-a2d16bc3256d" (UID: "26466126-8cc0-4784-9c40-a2d16bc3256d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:18.708199 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.708090 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-token" pod "26466126-8cc0-4784-9c40-a2d16bc3256d" (UID: "26466126-8cc0-4784-9c40-a2d16bc3256d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:18.813575 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.813106 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:18.813575 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.813172 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:18.813575 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.813187 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26466126-8cc0-4784-9c40-a2d16bc3256d-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:18.813575 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.813201 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrrct\" (UniqueName: \"kubernetes.io/projected/26466126-8cc0-4784-9c40-a2d16bc3256d-kube-api-access-xrrct\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:18.813575 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.813217 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26466126-8cc0-4784-9c40-a2d16bc3256d-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:18.813575 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:18.813233 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26466126-8cc0-4784-9c40-a2d16bc3256d-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:19.225521 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.225482 4531 generic.go:356] "Generic (PLEG): container finished" podID="26466126-8cc0-4784-9c40-a2d16bc3256d" containerID="6f425a94e5c2ae0ef2a8eff0adf72cc34402be61053ee1b9d85e4c3908d95676" exitCode=0 Oct 30 19:05:19.225717 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.225537 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" Oct 30 19:05:19.225717 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.225568 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" event={"ID":"26466126-8cc0-4784-9c40-a2d16bc3256d","Type":"ContainerDied","Data":"6f425a94e5c2ae0ef2a8eff0adf72cc34402be61053ee1b9d85e4c3908d95676"} Oct 30 19:05:19.225717 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.225632 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779" event={"ID":"26466126-8cc0-4784-9c40-a2d16bc3256d","Type":"ContainerDied","Data":"413a6057ef03cee8481b066e3ba86b860b4c07629ec226661ec90b07d70229d1"} Oct 30 19:05:19.225717 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.225660 4531 scope.go:122] "RemoveContainer" containerID="6f425a94e5c2ae0ef2a8eff0adf72cc34402be61053ee1b9d85e4c3908d95676" Oct 30 19:05:19.237021 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.236991 4531 scope.go:122] "RemoveContainer" containerID="6f425a94e5c2ae0ef2a8eff0adf72cc34402be61053ee1b9d85e4c3908d95676" Oct 30 19:05:19.237517 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:05:19.237480 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f425a94e5c2ae0ef2a8eff0adf72cc34402be61053ee1b9d85e4c3908d95676\": container with ID starting with 6f425a94e5c2ae0ef2a8eff0adf72cc34402be61053ee1b9d85e4c3908d95676 not found: ID does not exist" containerID="6f425a94e5c2ae0ef2a8eff0adf72cc34402be61053ee1b9d85e4c3908d95676" Oct 30 19:05:19.237616 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.237530 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f425a94e5c2ae0ef2a8eff0adf72cc34402be61053ee1b9d85e4c3908d95676"} err="failed to get container status \"6f425a94e5c2ae0ef2a8eff0adf72cc34402be61053ee1b9d85e4c3908d95676\": rpc error: code = NotFound desc = could not find container \"6f425a94e5c2ae0ef2a8eff0adf72cc34402be61053ee1b9d85e4c3908d95676\": container with ID starting with 6f425a94e5c2ae0ef2a8eff0adf72cc34402be61053ee1b9d85e4c3908d95676 not found: ID does not exist" Oct 30 19:05:19.391516 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.391478 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779"] Oct 30 19:05:19.395518 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.395483 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-7c779"] Oct 30 19:05:19.880583 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.880537 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6"] Oct 30 19:05:19.881076 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881059 4531 cpu_manager.go:414] "containerMap: removing container" podUID="52581cb3-251f-4450-b981-1b7fa19d9367" containerName="istio-proxy" Oct 30 19:05:19.881185 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881079 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52581cb3-251f-4450-b981-1b7fa19d9367" containerName="istio-proxy" Oct 30 19:05:19.881185 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881094 4531 cpu_manager.go:414] "containerMap: removing container" podUID="2ed7421d-881e-4a42-a128-e4852abe4cfd" containerName="istio-proxy" Oct 30 19:05:19.881185 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881100 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ed7421d-881e-4a42-a128-e4852abe4cfd" containerName="istio-proxy" Oct 30 19:05:19.881185 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881109 4531 cpu_manager.go:414] "containerMap: removing container" podUID="c3aece38-123d-42a2-a192-298b1d6b1579" containerName="payload-pre-processing" Oct 30 19:05:19.881185 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881115 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3aece38-123d-42a2-a192-298b1d6b1579" containerName="payload-pre-processing" Oct 30 19:05:19.881185 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881138 4531 cpu_manager.go:414] "containerMap: removing container" podUID="dfe944db-5201-4af3-8bb1-8e616ff90b00" containerName="payload-processing" Oct 30 19:05:19.881185 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881145 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfe944db-5201-4af3-8bb1-8e616ff90b00" containerName="payload-processing" Oct 30 19:05:19.881185 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881163 4531 cpu_manager.go:414] "containerMap: removing container" podUID="90016c74-97f8-4976-87c0-1603b897c2f3" containerName="istio-proxy" Oct 30 19:05:19.881185 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881169 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90016c74-97f8-4976-87c0-1603b897c2f3" containerName="istio-proxy" Oct 30 19:05:19.881185 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881185 4531 cpu_manager.go:414] "containerMap: removing container" podUID="52581cb3-251f-4450-b981-1b7fa19d9367" containerName="istio-proxy" Oct 30 19:05:19.881534 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881191 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52581cb3-251f-4450-b981-1b7fa19d9367" containerName="istio-proxy" Oct 30 19:05:19.881534 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881202 4531 cpu_manager.go:414] "containerMap: removing container" podUID="26466126-8cc0-4784-9c40-a2d16bc3256d" containerName="istio-proxy" Oct 30 19:05:19.881534 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881208 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26466126-8cc0-4784-9c40-a2d16bc3256d" containerName="istio-proxy" Oct 30 19:05:19.881534 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881224 4531 cpu_manager.go:414] "containerMap: removing container" podUID="0005aee3-3a38-431e-b86c-d1c40bab8483" containerName="istio-proxy" Oct 30 19:05:19.881534 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881231 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0005aee3-3a38-431e-b86c-d1c40bab8483" containerName="istio-proxy" Oct 30 19:05:19.881534 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881261 4531 cpu_manager.go:414] "containerMap: removing container" podUID="58ca32e1-9bcb-4600-b2ad-17e3656c4db6" containerName="revoke-keys" Oct 30 19:05:19.881534 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881267 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58ca32e1-9bcb-4600-b2ad-17e3656c4db6" containerName="revoke-keys" Oct 30 19:05:19.881534 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881278 4531 cpu_manager.go:414] "containerMap: removing container" podUID="b7ea737e-8a33-495c-959e-a037aed04436" containerName="istio-proxy" Oct 30 19:05:19.881534 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881285 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7ea737e-8a33-495c-959e-a037aed04436" containerName="istio-proxy" Oct 30 19:05:19.881534 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881390 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ed7421d-881e-4a42-a128-e4852abe4cfd" containerName="istio-proxy" Oct 30 19:05:19.881534 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881401 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3aece38-123d-42a2-a192-298b1d6b1579" containerName="payload-pre-processing" Oct 30 19:05:19.881534 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881407 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfe944db-5201-4af3-8bb1-8e616ff90b00" containerName="payload-processing" Oct 30 19:05:19.881534 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881416 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="90016c74-97f8-4976-87c0-1603b897c2f3" containerName="istio-proxy" Oct 30 19:05:19.881534 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881422 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="52581cb3-251f-4450-b981-1b7fa19d9367" containerName="istio-proxy" Oct 30 19:05:19.881534 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881427 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="26466126-8cc0-4784-9c40-a2d16bc3256d" containerName="istio-proxy" Oct 30 19:05:19.881534 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881433 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0005aee3-3a38-431e-b86c-d1c40bab8483" containerName="istio-proxy" Oct 30 19:05:19.881534 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881441 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="58ca32e1-9bcb-4600-b2ad-17e3656c4db6" containerName="revoke-keys" Oct 30 19:05:19.881534 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.881447 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7ea737e-8a33-495c-959e-a037aed04436" containerName="istio-proxy" Oct 30 19:05:19.886160 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.886097 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6" Oct 30 19:05:19.928699 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.928652 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78n7k\" (UniqueName: \"kubernetes.io/projected/ee3eed8b-8ad9-4770-ae08-8229037e36d7-kube-api-access-78n7k\") pod \"payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6\" (UID: \"ee3eed8b-8ad9-4770-ae08-8229037e36d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6" Oct 30 19:05:19.928892 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.928722 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee3eed8b-8ad9-4770-ae08-8229037e36d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6\" (UID: \"ee3eed8b-8ad9-4770-ae08-8229037e36d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6" Oct 30 19:05:19.941538 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.941496 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0c9a2b\"" Oct 30 19:05:19.941765 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.941653 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0c9a2b-dockercfg-fnn69\"" Oct 30 19:05:19.975298 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:19.974217 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6"] Oct 30 19:05:20.049031 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:20.048983 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78n7k\" (UniqueName: \"kubernetes.io/projected/ee3eed8b-8ad9-4770-ae08-8229037e36d7-kube-api-access-78n7k\") pod \"payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6\" (UID: \"ee3eed8b-8ad9-4770-ae08-8229037e36d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6" Oct 30 19:05:20.049314 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:20.049081 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee3eed8b-8ad9-4770-ae08-8229037e36d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6\" (UID: \"ee3eed8b-8ad9-4770-ae08-8229037e36d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6" Oct 30 19:05:20.053450 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:20.050392 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee3eed8b-8ad9-4770-ae08-8229037e36d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6\" (UID: \"ee3eed8b-8ad9-4770-ae08-8229037e36d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6" Oct 30 19:05:20.060970 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:20.060928 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p"] Oct 30 19:05:20.070949 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:20.064783 4531 cpu_manager.go:414] "containerMap: removing container" podUID="90016c74-97f8-4976-87c0-1603b897c2f3" containerName="istio-proxy" Oct 30 19:05:20.070949 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:20.064828 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90016c74-97f8-4976-87c0-1603b897c2f3" containerName="istio-proxy" Oct 30 19:05:20.070949 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:20.065216 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="52581cb3-251f-4450-b981-1b7fa19d9367" containerName="istio-proxy" Oct 30 19:05:20.070949 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:20.065256 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="90016c74-97f8-4976-87c0-1603b897c2f3" containerName="istio-proxy" Oct 30 19:05:20.079346 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:20.077471 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p" Oct 30 19:05:20.099857 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:20.099820 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26466126-8cc0-4784-9c40-a2d16bc3256d" path="/var/lib/kubelet/pods/26466126-8cc0-4784-9c40-a2d16bc3256d/volumes" Oct 30 19:05:20.102439 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:20.102400 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p"] Oct 30 19:05:20.115645 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:20.115606 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78n7k\" (UniqueName: \"kubernetes.io/projected/ee3eed8b-8ad9-4770-ae08-8229037e36d7-kube-api-access-78n7k\") pod \"payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6\" (UID: \"ee3eed8b-8ad9-4770-ae08-8229037e36d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6" Oct 30 19:05:20.200610 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:20.200519 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6" Oct 30 19:05:20.252413 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:20.252372 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5c113b-28b5-4ad8-aa77-b9ea1a733365-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p\" (UID: \"ef5c113b-28b5-4ad8-aa77-b9ea1a733365\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p" Oct 30 19:05:20.252604 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:20.252511 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zmvg\" (UniqueName: \"kubernetes.io/projected/ef5c113b-28b5-4ad8-aa77-b9ea1a733365-kube-api-access-4zmvg\") pod \"payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p\" (UID: \"ef5c113b-28b5-4ad8-aa77-b9ea1a733365\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p" Oct 30 19:05:20.353994 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:20.353947 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5c113b-28b5-4ad8-aa77-b9ea1a733365-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p\" (UID: \"ef5c113b-28b5-4ad8-aa77-b9ea1a733365\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p" Oct 30 19:05:20.354259 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:20.354032 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zmvg\" (UniqueName: \"kubernetes.io/projected/ef5c113b-28b5-4ad8-aa77-b9ea1a733365-kube-api-access-4zmvg\") pod \"payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p\" (UID: \"ef5c113b-28b5-4ad8-aa77-b9ea1a733365\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p" Oct 30 19:05:20.354765 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:20.354740 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5c113b-28b5-4ad8-aa77-b9ea1a733365-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p\" (UID: \"ef5c113b-28b5-4ad8-aa77-b9ea1a733365\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p" Oct 30 19:05:20.415292 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:20.415252 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zmvg\" (UniqueName: \"kubernetes.io/projected/ef5c113b-28b5-4ad8-aa77-b9ea1a733365-kube-api-access-4zmvg\") pod \"payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p\" (UID: \"ef5c113b-28b5-4ad8-aa77-b9ea1a733365\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p" Oct 30 19:05:20.587227 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:20.587123 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6"] Oct 30 19:05:20.590513 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:05:20.590462 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3eed8b_8ad9_4770_ae08_8229037e36d7.slice/crio-cadeba716ce2e256dd29f2c762dd640f9e481c3b10535fbdd9e535adf0185fdd WatchSource:0}: Error finding container cadeba716ce2e256dd29f2c762dd640f9e481c3b10535fbdd9e535adf0185fdd: Status 404 returned error can't find the container with id cadeba716ce2e256dd29f2c762dd640f9e481c3b10535fbdd9e535adf0185fdd Oct 30 19:05:20.695259 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:20.695223 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p" Oct 30 19:05:21.084113 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:21.084080 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p"] Oct 30 19:05:21.087199 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:05:21.087155 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5c113b_28b5_4ad8_aa77_b9ea1a733365.slice/crio-57175a08a8d07dd3fed35c64e4d7585929d3b13ef83e059a75c5c098a62c2c86 WatchSource:0}: Error finding container 57175a08a8d07dd3fed35c64e4d7585929d3b13ef83e059a75c5c098a62c2c86: Status 404 returned error can't find the container with id 57175a08a8d07dd3fed35c64e4d7585929d3b13ef83e059a75c5c098a62c2c86 Oct 30 19:05:21.244534 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:21.244434 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6" event={"ID":"ee3eed8b-8ad9-4770-ae08-8229037e36d7","Type":"ContainerStarted","Data":"89a857c0273bde7aa48f2afb7a47114ed7ec4574feb22eb2ca500c17080a2fb7"} Oct 30 19:05:21.244757 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:21.244625 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6" Oct 30 19:05:21.244757 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:21.244670 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6" event={"ID":"ee3eed8b-8ad9-4770-ae08-8229037e36d7","Type":"ContainerStarted","Data":"cadeba716ce2e256dd29f2c762dd640f9e481c3b10535fbdd9e535adf0185fdd"} Oct 30 19:05:21.247019 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:21.246975 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p" event={"ID":"ef5c113b-28b5-4ad8-aa77-b9ea1a733365","Type":"ContainerStarted","Data":"5c45f072730c0df21f36b7385e5b2a10f738f1fb77ed7572bb443319e44fcc2d"} Oct 30 19:05:21.247224 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:21.247047 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p" Oct 30 19:05:21.247224 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:21.247071 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p" event={"ID":"ef5c113b-28b5-4ad8-aa77-b9ea1a733365","Type":"ContainerStarted","Data":"57175a08a8d07dd3fed35c64e4d7585929d3b13ef83e059a75c5c098a62c2c86"} Oct 30 19:05:21.361807 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:21.361678 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6" podStartSLOduration=2.361661636 podStartE2EDuration="2.361661636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:21.35908893 +0000 UTC m=+1881.386193899" watchObservedRunningTime="2026-10-30 19:05:21.361661636 +0000 UTC m=+1881.388766594" Oct 30 19:05:21.474982 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:21.474911 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p" podStartSLOduration=2.474889929 podStartE2EDuration="2.474889929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:21.473769397 +0000 UTC m=+1881.500874357" watchObservedRunningTime="2026-10-30 19:05:21.474889929 +0000 UTC m=+1881.501994891" Oct 30 19:05:21.680219 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:21.680102 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:05:21.680219 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:21.680199 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" podUID="56bbae65-edbd-48ad-b1f6-7c5da8d2d53d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:05:22.690866 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.690832 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 19:05:22.786283 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.786243 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-envoy\") pod \"4adfad0a-5914-442f-afe1-c7f9c2f70849\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " Oct 30 19:05:22.786283 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.786287 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-podinfo\") pod \"4adfad0a-5914-442f-afe1-c7f9c2f70849\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " Oct 30 19:05:22.786575 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.786313 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-token\" (UniqueName: \"kubernetes.io/projected/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-token\") pod \"4adfad0a-5914-442f-afe1-c7f9c2f70849\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " Oct 30 19:05:22.786575 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.786345 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-workload-certs\") pod \"4adfad0a-5914-442f-afe1-c7f9c2f70849\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " Oct 30 19:05:22.786575 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.786455 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-credential-socket\") pod \"4adfad0a-5914-442f-afe1-c7f9c2f70849\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " Oct 30 19:05:22.786725 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.786603 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-data\") pod \"4adfad0a-5914-442f-afe1-c7f9c2f70849\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " Oct 30 19:05:22.786725 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.786647 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-workload-certs" pod "4adfad0a-5914-442f-afe1-c7f9c2f70849" (UID: "4adfad0a-5914-442f-afe1-c7f9c2f70849"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:22.786725 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.786668 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4adfad0a-5914-442f-afe1-c7f9c2f70849-kube-api-access-pfs7g\" (UniqueName: \"kubernetes.io/projected/4adfad0a-5914-442f-afe1-c7f9c2f70849-kube-api-access-pfs7g\") pod \"4adfad0a-5914-442f-afe1-c7f9c2f70849\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " Oct 30 19:05:22.786725 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.786682 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-credential-socket" pod "4adfad0a-5914-442f-afe1-c7f9c2f70849" (UID: "4adfad0a-5914-442f-afe1-c7f9c2f70849"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:22.786917 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.786804 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-data" pod "4adfad0a-5914-442f-afe1-c7f9c2f70849" (UID: "4adfad0a-5914-442f-afe1-c7f9c2f70849"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:22.786917 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.786822 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4adfad0a-5914-442f-afe1-c7f9c2f70849-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4adfad0a-5914-442f-afe1-c7f9c2f70849-istiod-ca-cert\") pod \"4adfad0a-5914-442f-afe1-c7f9c2f70849\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " Oct 30 19:05:22.786917 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.786867 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-workload-socket\") pod \"4adfad0a-5914-442f-afe1-c7f9c2f70849\" (UID: \"4adfad0a-5914-442f-afe1-c7f9c2f70849\") " Oct 30 19:05:22.787109 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.787088 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-workload-socket" pod "4adfad0a-5914-442f-afe1-c7f9c2f70849" (UID: "4adfad0a-5914-442f-afe1-c7f9c2f70849"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:22.787400 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.787369 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4adfad0a-5914-442f-afe1-c7f9c2f70849-istiod-ca-cert" pod "4adfad0a-5914-442f-afe1-c7f9c2f70849" (UID: "4adfad0a-5914-442f-afe1-c7f9c2f70849"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:22.787648 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.787599 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4adfad0a-5914-442f-afe1-c7f9c2f70849-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:22.787648 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.787620 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:22.787648 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.787636 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:22.787648 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.787652 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:22.787888 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.787667 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:22.788948 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.788918 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-envoy" pod "4adfad0a-5914-442f-afe1-c7f9c2f70849" (UID: "4adfad0a-5914-442f-afe1-c7f9c2f70849"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:22.789052 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.788953 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-token" pod "4adfad0a-5914-442f-afe1-c7f9c2f70849" (UID: "4adfad0a-5914-442f-afe1-c7f9c2f70849"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:22.789117 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.789105 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-podinfo" pod "4adfad0a-5914-442f-afe1-c7f9c2f70849" (UID: "4adfad0a-5914-442f-afe1-c7f9c2f70849"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:05:22.789325 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.789304 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4adfad0a-5914-442f-afe1-c7f9c2f70849-kube-api-access-pfs7g" pod "4adfad0a-5914-442f-afe1-c7f9c2f70849" (UID: "4adfad0a-5914-442f-afe1-c7f9c2f70849"). InnerVolumeSpecName "kube-api-access-pfs7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:22.889449 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.889404 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfs7g\" (UniqueName: \"kubernetes.io/projected/4adfad0a-5914-442f-afe1-c7f9c2f70849-kube-api-access-pfs7g\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:22.889449 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.889444 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:22.889449 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.889454 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:22.889449 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:22.889462 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4adfad0a-5914-442f-afe1-c7f9c2f70849-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:23.061819 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.061723 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:05:23.061819 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.061802 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" podUID="8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:05:23.260898 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.260862 4531 generic.go:356] "Generic (PLEG): container finished" podID="4adfad0a-5914-442f-afe1-c7f9c2f70849" containerID="8b5a37b986be2c06b49f1d324cea81721b09431bc76379c7d7d3cee807f7eb19" exitCode=0 Oct 30 19:05:23.261119 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.260931 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" Oct 30 19:05:23.261119 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.260950 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" event={"ID":"4adfad0a-5914-442f-afe1-c7f9c2f70849","Type":"ContainerDied","Data":"8b5a37b986be2c06b49f1d324cea81721b09431bc76379c7d7d3cee807f7eb19"} Oct 30 19:05:23.261119 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.261010 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q" event={"ID":"4adfad0a-5914-442f-afe1-c7f9c2f70849","Type":"ContainerDied","Data":"15b425da6db2f5b6632d79cc5810df5225906f3782eb1ec86f13cc9d5ab067cb"} Oct 30 19:05:23.261119 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.261033 4531 scope.go:122] "RemoveContainer" containerID="8b5a37b986be2c06b49f1d324cea81721b09431bc76379c7d7d3cee807f7eb19" Oct 30 19:05:23.271876 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.271849 4531 scope.go:122] "RemoveContainer" containerID="8b5a37b986be2c06b49f1d324cea81721b09431bc76379c7d7d3cee807f7eb19" Oct 30 19:05:23.272264 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:05:23.272236 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b5a37b986be2c06b49f1d324cea81721b09431bc76379c7d7d3cee807f7eb19\": container with ID starting with 8b5a37b986be2c06b49f1d324cea81721b09431bc76379c7d7d3cee807f7eb19 not found: ID does not exist" containerID="8b5a37b986be2c06b49f1d324cea81721b09431bc76379c7d7d3cee807f7eb19" Oct 30 19:05:23.272332 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.272275 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b5a37b986be2c06b49f1d324cea81721b09431bc76379c7d7d3cee807f7eb19"} err="failed to get container status \"8b5a37b986be2c06b49f1d324cea81721b09431bc76379c7d7d3cee807f7eb19\": rpc error: code = NotFound desc = could not find container \"8b5a37b986be2c06b49f1d324cea81721b09431bc76379c7d7d3cee807f7eb19\": container with ID starting with 8b5a37b986be2c06b49f1d324cea81721b09431bc76379c7d7d3cee807f7eb19 not found: ID does not exist" Oct 30 19:05:23.430788 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.430748 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q"] Oct 30 19:05:23.433253 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.433226 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-jk87q"] Oct 30 19:05:23.566854 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.566834 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 19:05:23.592399 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.592334 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 19:05:23.597815 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.597791 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-token\" (UniqueName: \"kubernetes.io/projected/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-token\") pod \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " Oct 30 19:05:23.597926 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.597877 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-credential-socket\") pod \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " Oct 30 19:05:23.597926 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.597906 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-token\" (UniqueName: \"kubernetes.io/projected/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-token\") pod \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " Oct 30 19:05:23.598030 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.597955 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istiod-ca-cert\") pod \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " Oct 30 19:05:23.598030 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.597989 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-kube-api-access-5wxck\" (UniqueName: \"kubernetes.io/projected/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-kube-api-access-5wxck\") pod \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " Oct 30 19:05:23.598153 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.598055 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-podinfo\") pod \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " Oct 30 19:05:23.598153 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.598106 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-data\") pod \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " Oct 30 19:05:23.598269 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.598168 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istiod-ca-cert\") pod \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " Oct 30 19:05:23.598269 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.598196 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-credential-socket" pod "56bbae65-edbd-48ad-b1f6-7c5da8d2d53d" (UID: "56bbae65-edbd-48ad-b1f6-7c5da8d2d53d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:23.598388 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.598315 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-envoy\") pod \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " Oct 30 19:05:23.598777 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.598662 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-data\") pod \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " Oct 30 19:05:23.598777 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.598435 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-data" pod "8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6" (UID: "8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:23.598777 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.598492 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istiod-ca-cert" pod "8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6" (UID: "8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:23.598777 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.598758 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-workload-certs\") pod \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " Oct 30 19:05:23.599023 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.598792 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-workload-certs\") pod \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " Oct 30 19:05:23.599023 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.598855 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-podinfo\") pod \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " Oct 30 19:05:23.599023 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.598886 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-workload-socket\") pod \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " Oct 30 19:05:23.599023 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.598928 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-workload-socket\") pod \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " Oct 30 19:05:23.599023 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.598970 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-envoy\") pod \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\" (UID: \"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d\") " Oct 30 19:05:23.599023 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.599009 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-credential-socket\") pod \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " Oct 30 19:05:23.599380 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.599055 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-kube-api-access-cv5p6\" (UniqueName: \"kubernetes.io/projected/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-kube-api-access-cv5p6\") pod \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\" (UID: \"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6\") " Oct 30 19:05:23.599895 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.599572 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:23.599895 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.599596 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:23.599895 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.599611 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:23.600937 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.598504 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istiod-ca-cert" pod "56bbae65-edbd-48ad-b1f6-7c5da8d2d53d" (UID: "56bbae65-edbd-48ad-b1f6-7c5da8d2d53d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:23.601040 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.598982 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-data" pod "56bbae65-edbd-48ad-b1f6-7c5da8d2d53d" (UID: "56bbae65-edbd-48ad-b1f6-7c5da8d2d53d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:23.601040 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.599622 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-workload-socket" pod "56bbae65-edbd-48ad-b1f6-7c5da8d2d53d" (UID: "56bbae65-edbd-48ad-b1f6-7c5da8d2d53d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:23.601040 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.600659 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-workload-certs" pod "56bbae65-edbd-48ad-b1f6-7c5da8d2d53d" (UID: "56bbae65-edbd-48ad-b1f6-7c5da8d2d53d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:23.601040 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.600908 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-workload-certs" pod "8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6" (UID: "8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:23.601040 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.600979 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-token" pod "8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6" (UID: "8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:23.601354 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.601167 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-workload-socket" pod "8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6" (UID: "8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:23.601354 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.601317 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-token" pod "56bbae65-edbd-48ad-b1f6-7c5da8d2d53d" (UID: "56bbae65-edbd-48ad-b1f6-7c5da8d2d53d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:23.601498 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.601476 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-credential-socket" pod "8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6" (UID: "8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:23.601805 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.601769 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-kube-api-access-5wxck" pod "56bbae65-edbd-48ad-b1f6-7c5da8d2d53d" (UID: "56bbae65-edbd-48ad-b1f6-7c5da8d2d53d"). InnerVolumeSpecName "kube-api-access-5wxck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:23.601891 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.601808 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-podinfo" pod "8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6" (UID: "8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:05:23.602097 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.602077 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-envoy" pod "8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6" (UID: "8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:23.602447 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.602427 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-podinfo" pod "56bbae65-edbd-48ad-b1f6-7c5da8d2d53d" (UID: "56bbae65-edbd-48ad-b1f6-7c5da8d2d53d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:05:23.603175 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.603120 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-envoy" pod "56bbae65-edbd-48ad-b1f6-7c5da8d2d53d" (UID: "56bbae65-edbd-48ad-b1f6-7c5da8d2d53d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:23.603401 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.603381 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-kube-api-access-cv5p6" pod "8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6" (UID: "8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6"). InnerVolumeSpecName "kube-api-access-cv5p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:23.700627 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.700592 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:23.700627 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.700625 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:23.700627 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.700636 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:23.701065 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.700644 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:23.701065 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.700657 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:23.701065 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.700671 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:23.701065 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.700680 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:23.701065 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.700689 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:23.701065 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.700697 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:23.701065 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.700704 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:23.701065 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.700712 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:23.701065 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.700720 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv5p6\" (UniqueName: \"kubernetes.io/projected/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-kube-api-access-cv5p6\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:23.701065 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.700730 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:23.701065 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.700737 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:23.701065 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:23.700745 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wxck\" (UniqueName: \"kubernetes.io/projected/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d-kube-api-access-5wxck\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:05:24.099392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:24.099271 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4adfad0a-5914-442f-afe1-c7f9c2f70849" path="/var/lib/kubelet/pods/4adfad0a-5914-442f-afe1-c7f9c2f70849/volumes" Oct 30 19:05:24.269246 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:24.269210 4531 generic.go:356] "Generic (PLEG): container finished" podID="56bbae65-edbd-48ad-b1f6-7c5da8d2d53d" containerID="305ec0fdca540995d5b873ff12e609e82d95ccb2bf3d18d015d7a60c2e8d95ef" exitCode=0 Oct 30 19:05:24.269444 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:24.269263 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" Oct 30 19:05:24.269444 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:24.269288 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" event={"ID":"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d","Type":"ContainerDied","Data":"305ec0fdca540995d5b873ff12e609e82d95ccb2bf3d18d015d7a60c2e8d95ef"} Oct 30 19:05:24.269444 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:24.269352 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265" event={"ID":"56bbae65-edbd-48ad-b1f6-7c5da8d2d53d","Type":"ContainerDied","Data":"dd4e0045b9d082b38c88cc06699c4f49f2128153dcb24dea0e646aebec02b78b"} Oct 30 19:05:24.269444 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:24.269391 4531 scope.go:122] "RemoveContainer" containerID="305ec0fdca540995d5b873ff12e609e82d95ccb2bf3d18d015d7a60c2e8d95ef" Oct 30 19:05:24.270700 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:24.270677 4531 generic.go:356] "Generic (PLEG): container finished" podID="8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6" containerID="bcd8a7d479e675dbf063838c670abb5163e72cd4035c52154867fede86c59f4d" exitCode=0 Oct 30 19:05:24.270816 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:24.270708 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" event={"ID":"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6","Type":"ContainerDied","Data":"bcd8a7d479e675dbf063838c670abb5163e72cd4035c52154867fede86c59f4d"} Oct 30 19:05:24.270816 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:24.270747 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" Oct 30 19:05:24.270816 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:24.270757 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh" event={"ID":"8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6","Type":"ContainerDied","Data":"0c0077a1d8c5283fb0bf221491efc15b7e0ba0879a9008cf88f2703357e09dd5"} Oct 30 19:05:24.279722 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:24.279703 4531 scope.go:122] "RemoveContainer" containerID="305ec0fdca540995d5b873ff12e609e82d95ccb2bf3d18d015d7a60c2e8d95ef" Oct 30 19:05:24.280044 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:05:24.280019 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"305ec0fdca540995d5b873ff12e609e82d95ccb2bf3d18d015d7a60c2e8d95ef\": container with ID starting with 305ec0fdca540995d5b873ff12e609e82d95ccb2bf3d18d015d7a60c2e8d95ef not found: ID does not exist" containerID="305ec0fdca540995d5b873ff12e609e82d95ccb2bf3d18d015d7a60c2e8d95ef" Oct 30 19:05:24.280118 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:24.280054 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"305ec0fdca540995d5b873ff12e609e82d95ccb2bf3d18d015d7a60c2e8d95ef"} err="failed to get container status \"305ec0fdca540995d5b873ff12e609e82d95ccb2bf3d18d015d7a60c2e8d95ef\": rpc error: code = NotFound desc = could not find container \"305ec0fdca540995d5b873ff12e609e82d95ccb2bf3d18d015d7a60c2e8d95ef\": container with ID starting with 305ec0fdca540995d5b873ff12e609e82d95ccb2bf3d18d015d7a60c2e8d95ef not found: ID does not exist" Oct 30 19:05:24.280118 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:24.280073 4531 scope.go:122] "RemoveContainer" containerID="bcd8a7d479e675dbf063838c670abb5163e72cd4035c52154867fede86c59f4d" Oct 30 19:05:24.292543 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:24.292385 4531 scope.go:122] "RemoveContainer" containerID="bcd8a7d479e675dbf063838c670abb5163e72cd4035c52154867fede86c59f4d" Oct 30 19:05:24.297797 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:05:24.297710 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcd8a7d479e675dbf063838c670abb5163e72cd4035c52154867fede86c59f4d\": container with ID starting with bcd8a7d479e675dbf063838c670abb5163e72cd4035c52154867fede86c59f4d not found: ID does not exist" containerID="bcd8a7d479e675dbf063838c670abb5163e72cd4035c52154867fede86c59f4d" Oct 30 19:05:24.298051 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:24.297764 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcd8a7d479e675dbf063838c670abb5163e72cd4035c52154867fede86c59f4d"} err="failed to get container status \"bcd8a7d479e675dbf063838c670abb5163e72cd4035c52154867fede86c59f4d\": rpc error: code = NotFound desc = could not find container \"bcd8a7d479e675dbf063838c670abb5163e72cd4035c52154867fede86c59f4d\": container with ID starting with bcd8a7d479e675dbf063838c670abb5163e72cd4035c52154867fede86c59f4d not found: ID does not exist" Oct 30 19:05:24.433876 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:24.433788 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265"] Oct 30 19:05:24.437508 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:24.437483 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-65265"] Oct 30 19:05:24.546351 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:24.546309 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh"] Oct 30 19:05:24.550378 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:24.550339 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-dlcdh"] Oct 30 19:05:26.099097 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:26.099051 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56bbae65-edbd-48ad-b1f6-7c5da8d2d53d" path="/var/lib/kubelet/pods/56bbae65-edbd-48ad-b1f6-7c5da8d2d53d/volumes" Oct 30 19:05:26.099588 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:26.099572 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6" path="/var/lib/kubelet/pods/8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6/volumes" Oct 30 19:05:26.403494 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:05:26.403401 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:05:31.971411 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:31.971368 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx"] Oct 30 19:05:31.971909 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:31.971893 4531 cpu_manager.go:414] "containerMap: removing container" podUID="4adfad0a-5914-442f-afe1-c7f9c2f70849" containerName="istio-proxy" Oct 30 19:05:31.971956 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:31.971911 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4adfad0a-5914-442f-afe1-c7f9c2f70849" containerName="istio-proxy" Oct 30 19:05:31.971956 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:31.971928 4531 cpu_manager.go:414] "containerMap: removing container" podUID="8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6" containerName="istio-proxy" Oct 30 19:05:31.971956 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:31.971937 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6" containerName="istio-proxy" Oct 30 19:05:31.971956 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:31.971954 4531 cpu_manager.go:414] "containerMap: removing container" podUID="56bbae65-edbd-48ad-b1f6-7c5da8d2d53d" containerName="istio-proxy" Oct 30 19:05:31.972083 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:31.971959 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56bbae65-edbd-48ad-b1f6-7c5da8d2d53d" containerName="istio-proxy" Oct 30 19:05:31.972083 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:31.972036 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4adfad0a-5914-442f-afe1-c7f9c2f70849" containerName="istio-proxy" Oct 30 19:05:31.972083 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:31.972046 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8192f408-5c6f-4c8e-b1b2-2729c7e4e8e6" containerName="istio-proxy" Oct 30 19:05:31.972083 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:31.972053 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="56bbae65-edbd-48ad-b1f6-7c5da8d2d53d" containerName="istio-proxy" Oct 30 19:05:31.976845 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:31.976819 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx" Oct 30 19:05:32.028929 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:32.028888 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0ee379\"" Oct 30 19:05:32.078433 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:32.078395 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0ee379-dockercfg-vbsgm\"" Oct 30 19:05:32.082807 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:32.082615 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx"] Oct 30 19:05:32.105236 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:32.105192 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wljrl\" (UniqueName: \"kubernetes.io/projected/4a126c43-8fec-4d89-b17e-9d34db70827f-kube-api-access-wljrl\") pod \"payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx\" (UID: \"4a126c43-8fec-4d89-b17e-9d34db70827f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx" Oct 30 19:05:32.105643 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:32.105615 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a126c43-8fec-4d89-b17e-9d34db70827f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx\" (UID: \"4a126c43-8fec-4d89-b17e-9d34db70827f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx" Oct 30 19:05:32.150498 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:32.150452 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn"] Oct 30 19:05:32.155826 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:32.155795 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn" Oct 30 19:05:32.172586 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:32.172548 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn"] Oct 30 19:05:32.207595 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:32.207544 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6599d4b0-0958-4800-be76-df33b1e67a87-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn\" (UID: \"6599d4b0-0958-4800-be76-df33b1e67a87\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn" Oct 30 19:05:32.207847 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:32.207820 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wljrl\" (UniqueName: \"kubernetes.io/projected/4a126c43-8fec-4d89-b17e-9d34db70827f-kube-api-access-wljrl\") pod \"payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx\" (UID: \"4a126c43-8fec-4d89-b17e-9d34db70827f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx" Oct 30 19:05:32.207946 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:32.207911 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhsfq\" (UniqueName: \"kubernetes.io/projected/6599d4b0-0958-4800-be76-df33b1e67a87-kube-api-access-mhsfq\") pod \"payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn\" (UID: \"6599d4b0-0958-4800-be76-df33b1e67a87\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn" Oct 30 19:05:32.208012 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:32.207946 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a126c43-8fec-4d89-b17e-9d34db70827f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx\" (UID: \"4a126c43-8fec-4d89-b17e-9d34db70827f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx" Oct 30 19:05:32.208764 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:32.208740 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a126c43-8fec-4d89-b17e-9d34db70827f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx\" (UID: \"4a126c43-8fec-4d89-b17e-9d34db70827f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx" Oct 30 19:05:32.257442 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:32.257406 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6" Oct 30 19:05:32.257714 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:32.257498 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p" Oct 30 19:05:32.269045 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:32.269008 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wljrl\" (UniqueName: \"kubernetes.io/projected/4a126c43-8fec-4d89-b17e-9d34db70827f-kube-api-access-wljrl\") pod \"payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx\" (UID: \"4a126c43-8fec-4d89-b17e-9d34db70827f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx" Oct 30 19:05:32.291303 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:32.291267 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx" Oct 30 19:05:32.310603 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:32.309809 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6599d4b0-0958-4800-be76-df33b1e67a87-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn\" (UID: \"6599d4b0-0958-4800-be76-df33b1e67a87\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn" Oct 30 19:05:32.310603 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:32.310078 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhsfq\" (UniqueName: \"kubernetes.io/projected/6599d4b0-0958-4800-be76-df33b1e67a87-kube-api-access-mhsfq\") pod \"payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn\" (UID: \"6599d4b0-0958-4800-be76-df33b1e67a87\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn" Oct 30 19:05:32.310603 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:32.310566 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6599d4b0-0958-4800-be76-df33b1e67a87-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn\" (UID: \"6599d4b0-0958-4800-be76-df33b1e67a87\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn" Oct 30 19:05:32.372673 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:32.372155 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhsfq\" (UniqueName: \"kubernetes.io/projected/6599d4b0-0958-4800-be76-df33b1e67a87-kube-api-access-mhsfq\") pod \"payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn\" (UID: \"6599d4b0-0958-4800-be76-df33b1e67a87\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn" Oct 30 19:05:32.491980 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:32.491931 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn" Oct 30 19:05:32.681983 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:32.681950 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx"] Oct 30 19:05:32.683602 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:05:32.683574 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a126c43_8fec_4d89_b17e_9d34db70827f.slice/crio-502a3f4720bccc6ed764c68a6a9a476c79c9f9b5033b8adee5656522521af81d WatchSource:0}: Error finding container 502a3f4720bccc6ed764c68a6a9a476c79c9f9b5033b8adee5656522521af81d: Status 404 returned error can't find the container with id 502a3f4720bccc6ed764c68a6a9a476c79c9f9b5033b8adee5656522521af81d Oct 30 19:05:32.882374 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:32.882343 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn"] Oct 30 19:05:32.884498 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:05:32.884463 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6599d4b0_0958_4800_be76_df33b1e67a87.slice/crio-1da77e5baa943340bc062527993ffe7178898710fa899412145458917f78a08c WatchSource:0}: Error finding container 1da77e5baa943340bc062527993ffe7178898710fa899412145458917f78a08c: Status 404 returned error can't find the container with id 1da77e5baa943340bc062527993ffe7178898710fa899412145458917f78a08c Oct 30 19:05:33.340943 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:33.340887 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn" event={"ID":"6599d4b0-0958-4800-be76-df33b1e67a87","Type":"ContainerStarted","Data":"ba34bcbe4abb90657f677d353b60a2a02d926883b235ea197a651db4f025eeed"} Oct 30 19:05:33.341437 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:33.341048 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn" event={"ID":"6599d4b0-0958-4800-be76-df33b1e67a87","Type":"ContainerStarted","Data":"1da77e5baa943340bc062527993ffe7178898710fa899412145458917f78a08c"} Oct 30 19:05:33.341437 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:33.341102 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn" Oct 30 19:05:33.342961 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:33.342925 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx" event={"ID":"4a126c43-8fec-4d89-b17e-9d34db70827f","Type":"ContainerStarted","Data":"a64978493f3f919f4e23c0745bd66927932be3c20b67367a45bfefa3d62d2f1f"} Oct 30 19:05:33.343100 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:33.342978 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx" event={"ID":"4a126c43-8fec-4d89-b17e-9d34db70827f","Type":"ContainerStarted","Data":"502a3f4720bccc6ed764c68a6a9a476c79c9f9b5033b8adee5656522521af81d"} Oct 30 19:05:33.343100 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:33.343007 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx" Oct 30 19:05:33.471245 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:33.471189 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn" podStartSLOduration=1.471174054 podStartE2EDuration="1.471174054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:33.470665294 +0000 UTC m=+1893.497770256" watchObservedRunningTime="2026-10-30 19:05:33.471174054 +0000 UTC m=+1893.498279013" Oct 30 19:05:33.601611 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:33.601449 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx" podStartSLOduration=2.601431314 podStartE2EDuration="2.601431314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:33.5975989 +0000 UTC m=+1893.624703870" watchObservedRunningTime="2026-10-30 19:05:33.601431314 +0000 UTC m=+1893.628536272" Oct 30 19:05:39.050352 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:39.050315 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f1ed4d8-ft2d5"] Oct 30 19:05:39.098908 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:39.098870 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f1ed4d8-ft2d5"] Oct 30 19:05:40.099419 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:40.099375 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6" path="/var/lib/kubelet/pods/6b90edb7-702f-45e4-b7d7-fdbe38ebfaf6/volumes" Oct 30 19:05:44.366196 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:44.360959 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn" Oct 30 19:05:44.376161 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:05:44.369161 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx" Oct 30 19:05:56.398900 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:05:56.398857 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:06:01.437728 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:01.437685 4531 scope.go:122] "RemoveContainer" containerID="fec45eca4353366acd32557d6bd821ed9b1831905b532ebe7a51dfeea04b5e52" Oct 30 19:06:05.221860 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:05.221816 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx"] Oct 30 19:06:05.227776 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:05.227747 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx" Oct 30 19:06:05.329702 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:05.329656 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/406b8a13-31e1-4660-a01a-c482d61f8fd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx\" (UID: \"406b8a13-31e1-4660-a01a-c482d61f8fd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx" Oct 30 19:06:05.329937 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:05.329795 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsnqx\" (UniqueName: \"kubernetes.io/projected/406b8a13-31e1-4660-a01a-c482d61f8fd1-kube-api-access-rsnqx\") pod \"payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx\" (UID: \"406b8a13-31e1-4660-a01a-c482d61f8fd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx" Oct 30 19:06:05.331026 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:05.330992 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a23af5\"" Oct 30 19:06:05.331241 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:05.331009 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a23af5-dockercfg-wtvjn\"" Oct 30 19:06:05.333909 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:05.333869 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx"] Oct 30 19:06:05.431473 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:05.431421 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsnqx\" (UniqueName: \"kubernetes.io/projected/406b8a13-31e1-4660-a01a-c482d61f8fd1-kube-api-access-rsnqx\") pod \"payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx\" (UID: \"406b8a13-31e1-4660-a01a-c482d61f8fd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx" Oct 30 19:06:05.431691 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:05.431599 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/406b8a13-31e1-4660-a01a-c482d61f8fd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx\" (UID: \"406b8a13-31e1-4660-a01a-c482d61f8fd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx" Oct 30 19:06:05.432357 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:05.432331 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/406b8a13-31e1-4660-a01a-c482d61f8fd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx\" (UID: \"406b8a13-31e1-4660-a01a-c482d61f8fd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx" Oct 30 19:06:05.493694 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:05.493661 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsnqx\" (UniqueName: \"kubernetes.io/projected/406b8a13-31e1-4660-a01a-c482d61f8fd1-kube-api-access-rsnqx\") pod \"payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx\" (UID: \"406b8a13-31e1-4660-a01a-c482d61f8fd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx" Oct 30 19:06:05.542414 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:05.542376 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx" Oct 30 19:06:05.923291 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:05.923059 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx"] Oct 30 19:06:05.926518 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:06:05.926487 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod406b8a13_31e1_4660_a01a_c482d61f8fd1.slice/crio-1b76714af5bf9d754f8519b9fac526e82600b75076865e2953b9d7bb136dc319 WatchSource:0}: Error finding container 1b76714af5bf9d754f8519b9fac526e82600b75076865e2953b9d7bb136dc319: Status 404 returned error can't find the container with id 1b76714af5bf9d754f8519b9fac526e82600b75076865e2953b9d7bb136dc319 Oct 30 19:06:05.929437 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:05.929411 4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:06:06.030324 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:06.030287 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf"] Oct 30 19:06:06.035264 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:06.035243 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf" Oct 30 19:06:06.100313 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:06.100282 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf"] Oct 30 19:06:06.138690 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:06.138639 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72725b3c-582d-44a2-9ca7-69fa4cfe2982-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf\" (UID: \"72725b3c-582d-44a2-9ca7-69fa4cfe2982\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf" Oct 30 19:06:06.138910 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:06.138709 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-749mw\" (UniqueName: \"kubernetes.io/projected/72725b3c-582d-44a2-9ca7-69fa4cfe2982-kube-api-access-749mw\") pod \"payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf\" (UID: \"72725b3c-582d-44a2-9ca7-69fa4cfe2982\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf" Oct 30 19:06:06.240343 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:06.240245 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72725b3c-582d-44a2-9ca7-69fa4cfe2982-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf\" (UID: \"72725b3c-582d-44a2-9ca7-69fa4cfe2982\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf" Oct 30 19:06:06.240343 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:06.240335 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-749mw\" (UniqueName: \"kubernetes.io/projected/72725b3c-582d-44a2-9ca7-69fa4cfe2982-kube-api-access-749mw\") pod \"payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf\" (UID: \"72725b3c-582d-44a2-9ca7-69fa4cfe2982\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf" Oct 30 19:06:06.241088 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:06.241068 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72725b3c-582d-44a2-9ca7-69fa4cfe2982-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf\" (UID: \"72725b3c-582d-44a2-9ca7-69fa4cfe2982\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf" Oct 30 19:06:06.301625 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:06.301594 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-749mw\" (UniqueName: \"kubernetes.io/projected/72725b3c-582d-44a2-9ca7-69fa4cfe2982-kube-api-access-749mw\") pod \"payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf\" (UID: \"72725b3c-582d-44a2-9ca7-69fa4cfe2982\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf" Oct 30 19:06:06.349680 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:06.349639 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf" Oct 30 19:06:06.559048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:06.559008 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx" event={"ID":"406b8a13-31e1-4660-a01a-c482d61f8fd1","Type":"ContainerStarted","Data":"1b76714af5bf9d754f8519b9fac526e82600b75076865e2953b9d7bb136dc319"} Oct 30 19:06:06.728766 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:06.728734 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf"] Oct 30 19:06:06.731365 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:06:06.731322 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72725b3c_582d_44a2_9ca7_69fa4cfe2982.slice/crio-1345b59bfcf420934cc37cbb4ea47dfdfdc2c041cfaaa87b60f394f069b1863b WatchSource:0}: Error finding container 1345b59bfcf420934cc37cbb4ea47dfdfdc2c041cfaaa87b60f394f069b1863b: Status 404 returned error can't find the container with id 1345b59bfcf420934cc37cbb4ea47dfdfdc2c041cfaaa87b60f394f069b1863b Oct 30 19:06:07.567530 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:07.567486 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf" event={"ID":"72725b3c-582d-44a2-9ca7-69fa4cfe2982","Type":"ContainerStarted","Data":"1345b59bfcf420934cc37cbb4ea47dfdfdc2c041cfaaa87b60f394f069b1863b"} Oct 30 19:06:09.604202 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:09.604111 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx" event={"ID":"406b8a13-31e1-4660-a01a-c482d61f8fd1","Type":"ContainerStarted","Data":"d7dca23bb80fb466735fe9bd8e0d825683193606def9dd0d91451e50782eab58"} Oct 30 19:06:09.604749 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:09.604236 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx" Oct 30 19:06:09.606232 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:09.606109 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf" event={"ID":"72725b3c-582d-44a2-9ca7-69fa4cfe2982","Type":"ContainerStarted","Data":"f903f726cbc900b0972772e1da6d1e037e928df3b009e46fe7d6542a02d7b40b"} Oct 30 19:06:09.606413 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:09.606253 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf" Oct 30 19:06:09.724705 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:09.724639 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx" podStartSLOduration=2.118057825 podStartE2EDuration="4.724619289s" totalImagesPullingTime="2.606561464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:09.724545361 +0000 UTC m=+1929.751650324" watchObservedRunningTime="2026-10-30 19:06:09.724619289 +0000 UTC m=+1929.751724248" Oct 30 19:06:09.851846 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:09.851353 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf" podStartSLOduration=3.036413454 podStartE2EDuration="4.851327124s" totalImagesPullingTime="1.81491367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:09.846954028 +0000 UTC m=+1929.874059007" watchObservedRunningTime="2026-10-30 19:06:09.851327124 +0000 UTC m=+1929.878432082" Oct 30 19:06:20.614753 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:20.614722 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx" Oct 30 19:06:20.615282 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:20.614793 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf" Oct 30 19:06:26.156589 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:26.156543 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96"] Oct 30 19:06:26.160391 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:26.160364 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96" Oct 30 19:06:26.212656 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:26.212624 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:06:26.263038 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:26.262862 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:06:26.263038 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:26.262870 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:06:26.263038 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:26.262937 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:06:26.265719 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:26.265684 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96"] Oct 30 19:06:26.303817 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:26.303774 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f83a517-0e78-40de-8808-e33beb798048-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96\" (UID: \"6f83a517-0e78-40de-8808-e33beb798048\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96" Oct 30 19:06:26.304012 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:26.303974 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f83a517-0e78-40de-8808-e33beb798048-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96\" (UID: \"6f83a517-0e78-40de-8808-e33beb798048\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96" Oct 30 19:06:26.304110 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:26.304090 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ghjm\" (UniqueName: \"kubernetes.io/projected/6f83a517-0e78-40de-8808-e33beb798048-kube-api-access-2ghjm\") pod \"maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96\" (UID: \"6f83a517-0e78-40de-8808-e33beb798048\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96" Oct 30 19:06:26.404667 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:06:26.404625 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:06:26.405104 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:26.405077 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f83a517-0e78-40de-8808-e33beb798048-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96\" (UID: \"6f83a517-0e78-40de-8808-e33beb798048\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96" Oct 30 19:06:26.405270 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:26.405250 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f83a517-0e78-40de-8808-e33beb798048-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96\" (UID: \"6f83a517-0e78-40de-8808-e33beb798048\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96" Oct 30 19:06:26.405339 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:26.405315 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ghjm\" (UniqueName: \"kubernetes.io/projected/6f83a517-0e78-40de-8808-e33beb798048-kube-api-access-2ghjm\") pod \"maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96\" (UID: \"6f83a517-0e78-40de-8808-e33beb798048\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96" Oct 30 19:06:26.406268 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:26.406240 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f83a517-0e78-40de-8808-e33beb798048-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96\" (UID: \"6f83a517-0e78-40de-8808-e33beb798048\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96" Oct 30 19:06:26.407944 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:26.407885 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f83a517-0e78-40de-8808-e33beb798048-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96\" (UID: \"6f83a517-0e78-40de-8808-e33beb798048\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96" Oct 30 19:06:26.469454 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:26.469417 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ghjm\" (UniqueName: \"kubernetes.io/projected/6f83a517-0e78-40de-8808-e33beb798048-kube-api-access-2ghjm\") pod \"maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96\" (UID: \"6f83a517-0e78-40de-8808-e33beb798048\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96" Oct 30 19:06:26.473976 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:26.473936 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96" Oct 30 19:06:26.865490 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:26.865461 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96"] Oct 30 19:06:26.867603 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:06:26.867572 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f83a517_0e78_40de_8808_e33beb798048.slice/crio-1c4fbb354f16674bed2b662987d22860dc56bc5e5440765531b4abf54b1ae588 WatchSource:0}: Error finding container 1c4fbb354f16674bed2b662987d22860dc56bc5e5440765531b4abf54b1ae588: Status 404 returned error can't find the container with id 1c4fbb354f16674bed2b662987d22860dc56bc5e5440765531b4abf54b1ae588 Oct 30 19:06:27.725576 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:27.725536 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96" event={"ID":"6f83a517-0e78-40de-8808-e33beb798048","Type":"ContainerStarted","Data":"c4cb4baf6e3e47ab912da1e2a511354dfc7f97229d9b7179a518224590ea3c32"} Oct 30 19:06:27.726005 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:27.725632 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96" event={"ID":"6f83a517-0e78-40de-8808-e33beb798048","Type":"ContainerStarted","Data":"1c4fbb354f16674bed2b662987d22860dc56bc5e5440765531b4abf54b1ae588"} Oct 30 19:06:27.842785 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:27.842725 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96" podStartSLOduration=1.842709095 podStartE2EDuration="1.842709095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:27.840372232 +0000 UTC m=+1947.867477191" watchObservedRunningTime="2026-10-30 19:06:27.842709095 +0000 UTC m=+1947.869814050" Oct 30 19:06:28.734265 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:28.734214 4531 generic.go:356] "Generic (PLEG): container finished" podID="6f83a517-0e78-40de-8808-e33beb798048" containerID="c4cb4baf6e3e47ab912da1e2a511354dfc7f97229d9b7179a518224590ea3c32" exitCode=0 Oct 30 19:06:28.734691 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:28.734280 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96" event={"ID":"6f83a517-0e78-40de-8808-e33beb798048","Type":"ContainerDied","Data":"c4cb4baf6e3e47ab912da1e2a511354dfc7f97229d9b7179a518224590ea3c32"} Oct 30 19:06:29.905963 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:29.905922 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96" Oct 30 19:06:29.955849 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:29.955802 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f83a517-0e78-40de-8808-e33beb798048-kube-api-access-2ghjm\" (UniqueName: \"kubernetes.io/projected/6f83a517-0e78-40de-8808-e33beb798048-kube-api-access-2ghjm\") pod \"6f83a517-0e78-40de-8808-e33beb798048\" (UID: \"6f83a517-0e78-40de-8808-e33beb798048\") " Oct 30 19:06:29.955849 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:29.955852 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f83a517-0e78-40de-8808-e33beb798048-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f83a517-0e78-40de-8808-e33beb798048-maas-api-xks-ca\") pod \"6f83a517-0e78-40de-8808-e33beb798048\" (UID: \"6f83a517-0e78-40de-8808-e33beb798048\") " Oct 30 19:06:29.956193 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:29.955903 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f83a517-0e78-40de-8808-e33beb798048-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f83a517-0e78-40de-8808-e33beb798048-maas-api-openshift-ca\") pod \"6f83a517-0e78-40de-8808-e33beb798048\" (UID: \"6f83a517-0e78-40de-8808-e33beb798048\") " Oct 30 19:06:29.956392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:29.956358 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f83a517-0e78-40de-8808-e33beb798048-maas-api-openshift-ca" pod "6f83a517-0e78-40de-8808-e33beb798048" (UID: "6f83a517-0e78-40de-8808-e33beb798048"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:06:29.958398 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:29.958362 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f83a517-0e78-40de-8808-e33beb798048-kube-api-access-2ghjm" pod "6f83a517-0e78-40de-8808-e33beb798048" (UID: "6f83a517-0e78-40de-8808-e33beb798048"). InnerVolumeSpecName "kube-api-access-2ghjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:06:29.958525 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:29.958499 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f83a517-0e78-40de-8808-e33beb798048-maas-api-xks-ca" pod "6f83a517-0e78-40de-8808-e33beb798048" (UID: "6f83a517-0e78-40de-8808-e33beb798048"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:06:30.057821 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:30.057729 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ghjm\" (UniqueName: \"kubernetes.io/projected/6f83a517-0e78-40de-8808-e33beb798048-kube-api-access-2ghjm\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:06:30.057821 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:30.057763 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f83a517-0e78-40de-8808-e33beb798048-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:06:30.057821 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:30.057773 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f83a517-0e78-40de-8808-e33beb798048-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:06:30.751304 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:30.751275 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96" Oct 30 19:06:30.751493 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:30.751316 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96" event={"ID":"6f83a517-0e78-40de-8808-e33beb798048","Type":"ContainerDied","Data":"1c4fbb354f16674bed2b662987d22860dc56bc5e5440765531b4abf54b1ae588"} Oct 30 19:06:30.751493 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:30.751366 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c4fbb354f16674bed2b662987d22860dc56bc5e5440765531b4abf54b1ae588" Oct 30 19:06:54.156492 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:54.156441 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96"] Oct 30 19:06:54.205350 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:54.205304 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-82305a-lnp96"] Oct 30 19:06:56.099422 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:06:56.099387 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f83a517-0e78-40de-8808-e33beb798048" path="/var/lib/kubelet/pods/6f83a517-0e78-40de-8808-e33beb798048/volumes" Oct 30 19:06:56.400677 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:06:56.400571 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:07:01.499614 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:01.499578 4531 scope.go:122] "RemoveContainer" containerID="74e06388324273e2ce5886265df6ff3c31108a25470ca3265053d93c43a1e813" Oct 30 19:07:01.510777 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:01.510747 4531 scope.go:122] "RemoveContainer" containerID="2ac379e71ba63f0d8be09f514d364c0597ab745d7c3f93b205a942d458d39e67" Oct 30 19:07:07.970065 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:07.970026 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh"] Oct 30 19:07:07.970610 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:07.970593 4531 cpu_manager.go:414] "containerMap: removing container" podUID="6f83a517-0e78-40de-8808-e33beb798048" containerName="revoke-keys" Oct 30 19:07:07.970655 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:07.970614 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f83a517-0e78-40de-8808-e33beb798048" containerName="revoke-keys" Oct 30 19:07:07.970709 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:07.970699 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f83a517-0e78-40de-8808-e33beb798048" containerName="revoke-keys" Oct 30 19:07:07.975497 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:07.975481 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh" Oct 30 19:07:08.021834 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:08.021795 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgzl9\" (UniqueName: \"kubernetes.io/projected/4b86ce37-2050-4fe1-9df5-63064867fc84-kube-api-access-fgzl9\") pod \"maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh\" (UID: \"4b86ce37-2050-4fe1-9df5-63064867fc84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh" Oct 30 19:07:08.022046 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:08.021854 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b86ce37-2050-4fe1-9df5-63064867fc84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh\" (UID: \"4b86ce37-2050-4fe1-9df5-63064867fc84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh" Oct 30 19:07:08.022046 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:08.022003 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b86ce37-2050-4fe1-9df5-63064867fc84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh\" (UID: \"4b86ce37-2050-4fe1-9df5-63064867fc84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh" Oct 30 19:07:08.029769 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:08.029733 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:07:08.034161 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:08.034111 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:07:08.034161 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:08.034113 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:07:08.036667 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:08.036640 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh"] Oct 30 19:07:08.079163 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:08.079093 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:07:08.123715 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:08.123680 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b86ce37-2050-4fe1-9df5-63064867fc84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh\" (UID: \"4b86ce37-2050-4fe1-9df5-63064867fc84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh" Oct 30 19:07:08.123896 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:08.123803 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgzl9\" (UniqueName: \"kubernetes.io/projected/4b86ce37-2050-4fe1-9df5-63064867fc84-kube-api-access-fgzl9\") pod \"maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh\" (UID: \"4b86ce37-2050-4fe1-9df5-63064867fc84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh" Oct 30 19:07:08.123896 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:08.123847 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b86ce37-2050-4fe1-9df5-63064867fc84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh\" (UID: \"4b86ce37-2050-4fe1-9df5-63064867fc84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh" Oct 30 19:07:08.124394 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:08.124369 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b86ce37-2050-4fe1-9df5-63064867fc84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh\" (UID: \"4b86ce37-2050-4fe1-9df5-63064867fc84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh" Oct 30 19:07:08.126574 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:08.126546 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b86ce37-2050-4fe1-9df5-63064867fc84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh\" (UID: \"4b86ce37-2050-4fe1-9df5-63064867fc84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh" Oct 30 19:07:08.187181 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:08.187115 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgzl9\" (UniqueName: \"kubernetes.io/projected/4b86ce37-2050-4fe1-9df5-63064867fc84-kube-api-access-fgzl9\") pod \"maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh\" (UID: \"4b86ce37-2050-4fe1-9df5-63064867fc84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh" Oct 30 19:07:08.288727 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:08.288693 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh" Oct 30 19:07:08.668491 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:08.668453 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh"] Oct 30 19:07:08.670515 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:07:08.670469 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b86ce37_2050_4fe1_9df5_63064867fc84.slice/crio-a199b9529363686631c0c012ff0dc22801fa56259c6e764465e0353dd91a8747 WatchSource:0}: Error finding container a199b9529363686631c0c012ff0dc22801fa56259c6e764465e0353dd91a8747: Status 404 returned error can't find the container with id a199b9529363686631c0c012ff0dc22801fa56259c6e764465e0353dd91a8747 Oct 30 19:07:09.001993 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:09.001955 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh" event={"ID":"4b86ce37-2050-4fe1-9df5-63064867fc84","Type":"ContainerStarted","Data":"1ee2d85a72dcba966e88dff5e31445096b49a7445ab163ca4c9a04683336e616"} Oct 30 19:07:09.002426 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:09.002012 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh" event={"ID":"4b86ce37-2050-4fe1-9df5-63064867fc84","Type":"ContainerStarted","Data":"a199b9529363686631c0c012ff0dc22801fa56259c6e764465e0353dd91a8747"} Oct 30 19:07:09.119485 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:09.119423 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh" podStartSLOduration=2.119405453 podStartE2EDuration="2.119405453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:09.11741161 +0000 UTC m=+1989.144516570" watchObservedRunningTime="2026-10-30 19:07:09.119405453 +0000 UTC m=+1989.146510411" Oct 30 19:07:10.009108 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:10.009071 4531 generic.go:356] "Generic (PLEG): container finished" podID="4b86ce37-2050-4fe1-9df5-63064867fc84" containerID="1ee2d85a72dcba966e88dff5e31445096b49a7445ab163ca4c9a04683336e616" exitCode=0 Oct 30 19:07:10.009560 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:10.009112 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh" event={"ID":"4b86ce37-2050-4fe1-9df5-63064867fc84","Type":"ContainerDied","Data":"1ee2d85a72dcba966e88dff5e31445096b49a7445ab163ca4c9a04683336e616"} Oct 30 19:07:11.166259 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:11.166233 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh" Oct 30 19:07:11.258466 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:11.258424 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b86ce37-2050-4fe1-9df5-63064867fc84-kube-api-access-fgzl9\" (UniqueName: \"kubernetes.io/projected/4b86ce37-2050-4fe1-9df5-63064867fc84-kube-api-access-fgzl9\") pod \"4b86ce37-2050-4fe1-9df5-63064867fc84\" (UID: \"4b86ce37-2050-4fe1-9df5-63064867fc84\") " Oct 30 19:07:11.258676 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:11.258530 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b86ce37-2050-4fe1-9df5-63064867fc84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b86ce37-2050-4fe1-9df5-63064867fc84-maas-api-openshift-ca\") pod \"4b86ce37-2050-4fe1-9df5-63064867fc84\" (UID: \"4b86ce37-2050-4fe1-9df5-63064867fc84\") " Oct 30 19:07:11.258676 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:11.258644 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b86ce37-2050-4fe1-9df5-63064867fc84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b86ce37-2050-4fe1-9df5-63064867fc84-maas-api-xks-ca\") pod \"4b86ce37-2050-4fe1-9df5-63064867fc84\" (UID: \"4b86ce37-2050-4fe1-9df5-63064867fc84\") " Oct 30 19:07:11.258936 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:11.258899 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b86ce37-2050-4fe1-9df5-63064867fc84-maas-api-openshift-ca" pod "4b86ce37-2050-4fe1-9df5-63064867fc84" (UID: "4b86ce37-2050-4fe1-9df5-63064867fc84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:07:11.259172 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:11.259117 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b86ce37-2050-4fe1-9df5-63064867fc84-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:07:11.260849 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:11.260823 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b86ce37-2050-4fe1-9df5-63064867fc84-maas-api-xks-ca" pod "4b86ce37-2050-4fe1-9df5-63064867fc84" (UID: "4b86ce37-2050-4fe1-9df5-63064867fc84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:07:11.260849 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:11.260843 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b86ce37-2050-4fe1-9df5-63064867fc84-kube-api-access-fgzl9" pod "4b86ce37-2050-4fe1-9df5-63064867fc84" (UID: "4b86ce37-2050-4fe1-9df5-63064867fc84"). InnerVolumeSpecName "kube-api-access-fgzl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:07:11.360411 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:11.360362 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgzl9\" (UniqueName: \"kubernetes.io/projected/4b86ce37-2050-4fe1-9df5-63064867fc84-kube-api-access-fgzl9\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:07:11.360411 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:11.360402 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b86ce37-2050-4fe1-9df5-63064867fc84-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:07:12.021970 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:12.021927 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh" event={"ID":"4b86ce37-2050-4fe1-9df5-63064867fc84","Type":"ContainerDied","Data":"a199b9529363686631c0c012ff0dc22801fa56259c6e764465e0353dd91a8747"} Oct 30 19:07:12.021970 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:12.021956 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh" Oct 30 19:07:12.022269 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:12.022010 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a199b9529363686631c0c012ff0dc22801fa56259c6e764465e0353dd91a8747" Oct 30 19:07:26.397862 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:07:26.397825 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:07:27.700463 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:27.700427 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh"] Oct 30 19:07:27.749658 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:27.749615 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fee9f0-jj5gh"] Oct 30 19:07:28.085654 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:28.085617 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7"] Oct 30 19:07:28.086079 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:28.086067 4531 cpu_manager.go:414] "containerMap: removing container" podUID="4b86ce37-2050-4fe1-9df5-63064867fc84" containerName="revoke-keys" Oct 30 19:07:28.086151 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:28.086080 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b86ce37-2050-4fe1-9df5-63064867fc84" containerName="revoke-keys" Oct 30 19:07:28.086206 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:28.086194 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b86ce37-2050-4fe1-9df5-63064867fc84" containerName="revoke-keys" Oct 30 19:07:28.089416 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:28.089398 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7" Oct 30 19:07:28.101091 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:28.101060 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b86ce37-2050-4fe1-9df5-63064867fc84" path="/var/lib/kubelet/pods/4b86ce37-2050-4fe1-9df5-63064867fc84/volumes" Oct 30 19:07:28.141175 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:28.141117 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:07:28.141375 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:28.141274 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:07:28.190791 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:28.190748 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:07:28.190791 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:28.190748 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:07:28.193583 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:28.193555 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7"] Oct 30 19:07:28.269980 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:28.269937 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2936362b-fefc-4625-bbfc-57da0fb1dae6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7\" (UID: \"2936362b-fefc-4625-bbfc-57da0fb1dae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7" Oct 30 19:07:28.270220 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:28.270015 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx8dx\" (UniqueName: \"kubernetes.io/projected/2936362b-fefc-4625-bbfc-57da0fb1dae6-kube-api-access-fx8dx\") pod \"maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7\" (UID: \"2936362b-fefc-4625-bbfc-57da0fb1dae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7" Oct 30 19:07:28.270220 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:28.270089 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2936362b-fefc-4625-bbfc-57da0fb1dae6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7\" (UID: \"2936362b-fefc-4625-bbfc-57da0fb1dae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7" Oct 30 19:07:28.371426 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:28.371324 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx8dx\" (UniqueName: \"kubernetes.io/projected/2936362b-fefc-4625-bbfc-57da0fb1dae6-kube-api-access-fx8dx\") pod \"maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7\" (UID: \"2936362b-fefc-4625-bbfc-57da0fb1dae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7" Oct 30 19:07:28.371426 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:28.371369 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2936362b-fefc-4625-bbfc-57da0fb1dae6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7\" (UID: \"2936362b-fefc-4625-bbfc-57da0fb1dae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7" Oct 30 19:07:28.371694 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:28.371520 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2936362b-fefc-4625-bbfc-57da0fb1dae6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7\" (UID: \"2936362b-fefc-4625-bbfc-57da0fb1dae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7" Oct 30 19:07:28.372172 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:28.372112 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2936362b-fefc-4625-bbfc-57da0fb1dae6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7\" (UID: \"2936362b-fefc-4625-bbfc-57da0fb1dae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7" Oct 30 19:07:28.374138 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:28.374102 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2936362b-fefc-4625-bbfc-57da0fb1dae6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7\" (UID: \"2936362b-fefc-4625-bbfc-57da0fb1dae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7" Oct 30 19:07:28.433342 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:28.433308 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx8dx\" (UniqueName: \"kubernetes.io/projected/2936362b-fefc-4625-bbfc-57da0fb1dae6-kube-api-access-fx8dx\") pod \"maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7\" (UID: \"2936362b-fefc-4625-bbfc-57da0fb1dae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7" Oct 30 19:07:28.704153 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:28.704037 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7" Oct 30 19:07:29.083615 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:29.083589 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7"] Oct 30 19:07:29.085805 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:07:29.085681 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2936362b_fefc_4625_bbfc_57da0fb1dae6.slice/crio-c17ea263e1f88dcf7d86e1ffacca8dbc4508888627d01f3660cb04ccd3b9f556 WatchSource:0}: Error finding container c17ea263e1f88dcf7d86e1ffacca8dbc4508888627d01f3660cb04ccd3b9f556: Status 404 returned error can't find the container with id c17ea263e1f88dcf7d86e1ffacca8dbc4508888627d01f3660cb04ccd3b9f556 Oct 30 19:07:29.126993 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:29.126964 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7" event={"ID":"2936362b-fefc-4625-bbfc-57da0fb1dae6","Type":"ContainerStarted","Data":"c17ea263e1f88dcf7d86e1ffacca8dbc4508888627d01f3660cb04ccd3b9f556"} Oct 30 19:07:30.134770 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:30.134731 4531 generic.go:356] "Generic (PLEG): container finished" podID="2936362b-fefc-4625-bbfc-57da0fb1dae6" containerID="1e58b95edf25fcc3734abbd560994f57e30dcbc5c494a6a8727c8bbecd2efc08" exitCode=0 Oct 30 19:07:30.135227 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:30.134795 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7" event={"ID":"2936362b-fefc-4625-bbfc-57da0fb1dae6","Type":"ContainerDied","Data":"1e58b95edf25fcc3734abbd560994f57e30dcbc5c494a6a8727c8bbecd2efc08"} Oct 30 19:07:31.288039 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:31.288012 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7" Oct 30 19:07:31.306026 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:31.305997 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2936362b-fefc-4625-bbfc-57da0fb1dae6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2936362b-fefc-4625-bbfc-57da0fb1dae6-maas-api-openshift-ca\") pod \"2936362b-fefc-4625-bbfc-57da0fb1dae6\" (UID: \"2936362b-fefc-4625-bbfc-57da0fb1dae6\") " Oct 30 19:07:31.306219 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:31.306057 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2936362b-fefc-4625-bbfc-57da0fb1dae6-kube-api-access-fx8dx\" (UniqueName: \"kubernetes.io/projected/2936362b-fefc-4625-bbfc-57da0fb1dae6-kube-api-access-fx8dx\") pod \"2936362b-fefc-4625-bbfc-57da0fb1dae6\" (UID: \"2936362b-fefc-4625-bbfc-57da0fb1dae6\") " Oct 30 19:07:31.306304 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:31.306232 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2936362b-fefc-4625-bbfc-57da0fb1dae6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2936362b-fefc-4625-bbfc-57da0fb1dae6-maas-api-xks-ca\") pod \"2936362b-fefc-4625-bbfc-57da0fb1dae6\" (UID: \"2936362b-fefc-4625-bbfc-57da0fb1dae6\") " Oct 30 19:07:31.306398 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:31.306374 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2936362b-fefc-4625-bbfc-57da0fb1dae6-maas-api-openshift-ca" pod "2936362b-fefc-4625-bbfc-57da0fb1dae6" (UID: "2936362b-fefc-4625-bbfc-57da0fb1dae6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:07:31.306649 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:31.306631 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2936362b-fefc-4625-bbfc-57da0fb1dae6-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:07:31.308394 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:31.308369 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2936362b-fefc-4625-bbfc-57da0fb1dae6-maas-api-xks-ca" pod "2936362b-fefc-4625-bbfc-57da0fb1dae6" (UID: "2936362b-fefc-4625-bbfc-57da0fb1dae6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:07:31.308494 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:31.308391 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2936362b-fefc-4625-bbfc-57da0fb1dae6-kube-api-access-fx8dx" pod "2936362b-fefc-4625-bbfc-57da0fb1dae6" (UID: "2936362b-fefc-4625-bbfc-57da0fb1dae6"). InnerVolumeSpecName "kube-api-access-fx8dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:07:31.408023 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:31.407933 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx8dx\" (UniqueName: \"kubernetes.io/projected/2936362b-fefc-4625-bbfc-57da0fb1dae6-kube-api-access-fx8dx\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:07:31.408023 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:31.407970 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2936362b-fefc-4625-bbfc-57da0fb1dae6-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:07:31.842078 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:31.842036 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p"] Oct 30 19:07:31.948557 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:31.948512 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6"] Oct 30 19:07:32.146466 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:32.146436 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7" Oct 30 19:07:32.146663 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:32.146442 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7" event={"ID":"2936362b-fefc-4625-bbfc-57da0fb1dae6","Type":"ContainerDied","Data":"c17ea263e1f88dcf7d86e1ffacca8dbc4508888627d01f3660cb04ccd3b9f556"} Oct 30 19:07:32.146663 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:32.146563 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c17ea263e1f88dcf7d86e1ffacca8dbc4508888627d01f3660cb04ccd3b9f556" Oct 30 19:07:36.859210 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:36.849570 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p" podUID="ef5c113b-28b5-4ad8-aa77-b9ea1a733365" containerName="payload-processing" containerID="cri-o://5c45f072730c0df21f36b7385e5b2a10f738f1fb77ed7572bb443319e44fcc2d" gracePeriod=25 Oct 30 19:07:36.949787 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:36.949730 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6" podUID="ee3eed8b-8ad9-4770-ae08-8229037e36d7" containerName="payload-pre-processing" containerID="cri-o://89a857c0273bde7aa48f2afb7a47114ed7ec4574feb22eb2ca500c17080a2fb7" gracePeriod=25 Oct 30 19:07:37.209612 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:37.209567 4531 generic.go:356] "Generic (PLEG): container finished" podID="ef5c113b-28b5-4ad8-aa77-b9ea1a733365" containerID="5c45f072730c0df21f36b7385e5b2a10f738f1fb77ed7572bb443319e44fcc2d" exitCode=0 Oct 30 19:07:37.209824 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:37.209770 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p" event={"ID":"ef5c113b-28b5-4ad8-aa77-b9ea1a733365","Type":"ContainerDied","Data":"5c45f072730c0df21f36b7385e5b2a10f738f1fb77ed7572bb443319e44fcc2d"} Oct 30 19:07:37.212658 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:37.212615 4531 generic.go:356] "Generic (PLEG): container finished" podID="ee3eed8b-8ad9-4770-ae08-8229037e36d7" containerID="89a857c0273bde7aa48f2afb7a47114ed7ec4574feb22eb2ca500c17080a2fb7" exitCode=0 Oct 30 19:07:37.212861 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:37.212714 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6" event={"ID":"ee3eed8b-8ad9-4770-ae08-8229037e36d7","Type":"ContainerDied","Data":"89a857c0273bde7aa48f2afb7a47114ed7ec4574feb22eb2ca500c17080a2fb7"} Oct 30 19:07:37.268308 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:37.268265 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p" Oct 30 19:07:37.304537 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:37.304493 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6" Oct 30 19:07:37.408828 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:37.408728 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee3eed8b-8ad9-4770-ae08-8229037e36d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee3eed8b-8ad9-4770-ae08-8229037e36d7-plugins-config-volume\") pod \"ee3eed8b-8ad9-4770-ae08-8229037e36d7\" (UID: \"ee3eed8b-8ad9-4770-ae08-8229037e36d7\") " Oct 30 19:07:37.409007 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:37.408866 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef5c113b-28b5-4ad8-aa77-b9ea1a733365-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5c113b-28b5-4ad8-aa77-b9ea1a733365-plugins-config-volume\") pod \"ef5c113b-28b5-4ad8-aa77-b9ea1a733365\" (UID: \"ef5c113b-28b5-4ad8-aa77-b9ea1a733365\") " Oct 30 19:07:37.409007 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:37.408897 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef5c113b-28b5-4ad8-aa77-b9ea1a733365-kube-api-access-4zmvg\" (UniqueName: \"kubernetes.io/projected/ef5c113b-28b5-4ad8-aa77-b9ea1a733365-kube-api-access-4zmvg\") pod \"ef5c113b-28b5-4ad8-aa77-b9ea1a733365\" (UID: \"ef5c113b-28b5-4ad8-aa77-b9ea1a733365\") " Oct 30 19:07:37.409104 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:37.408992 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee3eed8b-8ad9-4770-ae08-8229037e36d7-kube-api-access-78n7k\" (UniqueName: \"kubernetes.io/projected/ee3eed8b-8ad9-4770-ae08-8229037e36d7-kube-api-access-78n7k\") pod \"ee3eed8b-8ad9-4770-ae08-8229037e36d7\" (UID: \"ee3eed8b-8ad9-4770-ae08-8229037e36d7\") " Oct 30 19:07:37.425992 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:37.423787 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee3eed8b-8ad9-4770-ae08-8229037e36d7-plugins-config-volume" pod "ee3eed8b-8ad9-4770-ae08-8229037e36d7" (UID: "ee3eed8b-8ad9-4770-ae08-8229037e36d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:07:37.425992 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:37.424867 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef5c113b-28b5-4ad8-aa77-b9ea1a733365-plugins-config-volume" pod "ef5c113b-28b5-4ad8-aa77-b9ea1a733365" (UID: "ef5c113b-28b5-4ad8-aa77-b9ea1a733365"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:07:37.432860 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:37.432811 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3eed8b-8ad9-4770-ae08-8229037e36d7-kube-api-access-78n7k" pod "ee3eed8b-8ad9-4770-ae08-8229037e36d7" (UID: "ee3eed8b-8ad9-4770-ae08-8229037e36d7"). InnerVolumeSpecName "kube-api-access-78n7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:07:37.435521 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:37.435442 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5c113b-28b5-4ad8-aa77-b9ea1a733365-kube-api-access-4zmvg" pod "ef5c113b-28b5-4ad8-aa77-b9ea1a733365" (UID: "ef5c113b-28b5-4ad8-aa77-b9ea1a733365"). InnerVolumeSpecName "kube-api-access-4zmvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:07:37.511096 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:37.511049 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee3eed8b-8ad9-4770-ae08-8229037e36d7-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:07:37.511096 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:37.511093 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5c113b-28b5-4ad8-aa77-b9ea1a733365-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:07:37.511965 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:37.511109 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zmvg\" (UniqueName: \"kubernetes.io/projected/ef5c113b-28b5-4ad8-aa77-b9ea1a733365-kube-api-access-4zmvg\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:07:37.511965 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:37.511123 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78n7k\" (UniqueName: \"kubernetes.io/projected/ee3eed8b-8ad9-4770-ae08-8229037e36d7-kube-api-access-78n7k\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:07:38.221565 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:38.221464 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p" Oct 30 19:07:38.221565 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:38.221478 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p" event={"ID":"ef5c113b-28b5-4ad8-aa77-b9ea1a733365","Type":"ContainerDied","Data":"57175a08a8d07dd3fed35c64e4d7585929d3b13ef83e059a75c5c098a62c2c86"} Oct 30 19:07:38.221565 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:38.221544 4531 scope.go:122] "RemoveContainer" containerID="5c45f072730c0df21f36b7385e5b2a10f738f1fb77ed7572bb443319e44fcc2d" Oct 30 19:07:38.224405 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:38.224149 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6" event={"ID":"ee3eed8b-8ad9-4770-ae08-8229037e36d7","Type":"ContainerDied","Data":"cadeba716ce2e256dd29f2c762dd640f9e481c3b10535fbdd9e535adf0185fdd"} Oct 30 19:07:38.224554 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:38.224438 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6" Oct 30 19:07:38.233338 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:38.233313 4531 scope.go:122] "RemoveContainer" containerID="89a857c0273bde7aa48f2afb7a47114ed7ec4574feb22eb2ca500c17080a2fb7" Oct 30 19:07:38.386307 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:38.386264 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6"] Oct 30 19:07:38.437113 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:38.437072 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-64c49bd959-kxzn6"] Oct 30 19:07:38.501031 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:38.500991 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p"] Oct 30 19:07:38.504000 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:38.503960 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-7cd76d86f4-g7n8p"] Oct 30 19:07:40.099515 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:40.099474 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee3eed8b-8ad9-4770-ae08-8229037e36d7" path="/var/lib/kubelet/pods/ee3eed8b-8ad9-4770-ae08-8229037e36d7/volumes" Oct 30 19:07:40.099939 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:40.099895 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef5c113b-28b5-4ad8-aa77-b9ea1a733365" path="/var/lib/kubelet/pods/ef5c113b-28b5-4ad8-aa77-b9ea1a733365/volumes" Oct 30 19:07:41.788118 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:41.788039 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf"] Oct 30 19:07:41.788555 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:41.788539 4531 cpu_manager.go:414] "containerMap: removing container" podUID="ee3eed8b-8ad9-4770-ae08-8229037e36d7" containerName="payload-pre-processing" Oct 30 19:07:41.788599 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:41.788557 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3eed8b-8ad9-4770-ae08-8229037e36d7" containerName="payload-pre-processing" Oct 30 19:07:41.788599 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:41.788583 4531 cpu_manager.go:414] "containerMap: removing container" podUID="2936362b-fefc-4625-bbfc-57da0fb1dae6" containerName="revoke-keys" Oct 30 19:07:41.788599 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:41.788590 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2936362b-fefc-4625-bbfc-57da0fb1dae6" containerName="revoke-keys" Oct 30 19:07:41.788695 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:41.788611 4531 cpu_manager.go:414] "containerMap: removing container" podUID="ef5c113b-28b5-4ad8-aa77-b9ea1a733365" containerName="payload-processing" Oct 30 19:07:41.788695 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:41.788616 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef5c113b-28b5-4ad8-aa77-b9ea1a733365" containerName="payload-processing" Oct 30 19:07:41.788695 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:41.788689 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee3eed8b-8ad9-4770-ae08-8229037e36d7" containerName="payload-pre-processing" Oct 30 19:07:41.788787 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:41.788701 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2936362b-fefc-4625-bbfc-57da0fb1dae6" containerName="revoke-keys" Oct 30 19:07:41.788787 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:41.788711 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef5c113b-28b5-4ad8-aa77-b9ea1a733365" containerName="payload-processing" Oct 30 19:07:41.793361 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:41.793342 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf" Oct 30 19:07:41.858286 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:41.858251 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf"] Oct 30 19:07:41.868725 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:41.867567 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f666e16-86ea-46f9-908f-48dcacfb331e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf\" (UID: \"8f666e16-86ea-46f9-908f-48dcacfb331e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf" Oct 30 19:07:41.868725 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:41.867757 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xscl\" (UniqueName: \"kubernetes.io/projected/8f666e16-86ea-46f9-908f-48dcacfb331e-kube-api-access-2xscl\") pod \"maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf\" (UID: \"8f666e16-86ea-46f9-908f-48dcacfb331e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf" Oct 30 19:07:41.868725 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:41.867828 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f666e16-86ea-46f9-908f-48dcacfb331e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf\" (UID: \"8f666e16-86ea-46f9-908f-48dcacfb331e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf" Oct 30 19:07:41.898247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:41.898209 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:07:41.898508 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:41.898472 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:07:41.898508 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:41.898339 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:07:41.898683 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:41.898265 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:07:41.905102 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:41.905053 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn"] Oct 30 19:07:41.968984 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:41.968932 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f666e16-86ea-46f9-908f-48dcacfb331e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf\" (UID: \"8f666e16-86ea-46f9-908f-48dcacfb331e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf" Oct 30 19:07:41.969229 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:41.969055 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xscl\" (UniqueName: \"kubernetes.io/projected/8f666e16-86ea-46f9-908f-48dcacfb331e-kube-api-access-2xscl\") pod \"maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf\" (UID: \"8f666e16-86ea-46f9-908f-48dcacfb331e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf" Oct 30 19:07:41.969229 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:41.969103 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f666e16-86ea-46f9-908f-48dcacfb331e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf\" (UID: \"8f666e16-86ea-46f9-908f-48dcacfb331e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf" Oct 30 19:07:41.969747 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:41.969717 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f666e16-86ea-46f9-908f-48dcacfb331e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf\" (UID: \"8f666e16-86ea-46f9-908f-48dcacfb331e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf" Oct 30 19:07:41.971798 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:41.971762 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f666e16-86ea-46f9-908f-48dcacfb331e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf\" (UID: \"8f666e16-86ea-46f9-908f-48dcacfb331e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf" Oct 30 19:07:42.014100 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:42.014057 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx"] Oct 30 19:07:42.051710 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:42.047185 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xscl\" (UniqueName: \"kubernetes.io/projected/8f666e16-86ea-46f9-908f-48dcacfb331e-kube-api-access-2xscl\") pod \"maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf\" (UID: \"8f666e16-86ea-46f9-908f-48dcacfb331e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf" Oct 30 19:07:42.106658 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:42.106616 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf" Oct 30 19:07:42.522263 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:42.522206 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf"] Oct 30 19:07:43.262120 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:43.262082 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf" event={"ID":"8f666e16-86ea-46f9-908f-48dcacfb331e","Type":"ContainerStarted","Data":"d0f8adc933bca5440734098d4d4a085dfd60916753b03a12dbe6b9898280f455"} Oct 30 19:07:43.262120 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:43.262150 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf" event={"ID":"8f666e16-86ea-46f9-908f-48dcacfb331e","Type":"ContainerStarted","Data":"48710578b8208111a008c534f5eaf1b2e4f3a52d14ed205a2275381a0c5f6cd0"} Oct 30 19:07:43.378832 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:43.378784 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf" podStartSLOduration=2.378768595 podStartE2EDuration="2.378768595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:43.377673487 +0000 UTC m=+2023.404778450" watchObservedRunningTime="2026-10-30 19:07:43.378768595 +0000 UTC m=+2023.405873556" Oct 30 19:07:43.429796 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:43.429768 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh"] Oct 30 19:07:43.433601 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:43.433582 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh" Oct 30 19:07:43.485741 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:43.485700 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1acf9b\"" Oct 30 19:07:43.486016 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:43.485996 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1acf9b-dockercfg-lpgsh\"" Oct 30 19:07:43.507458 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:43.500797 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh"] Oct 30 19:07:43.551775 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:43.551669 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6977f362-678e-4e26-bcc0-1e38c063fca5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh\" (UID: \"6977f362-678e-4e26-bcc0-1e38c063fca5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh" Oct 30 19:07:43.551775 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:43.551752 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgmrs\" (UniqueName: \"kubernetes.io/projected/6977f362-678e-4e26-bcc0-1e38c063fca5-kube-api-access-fgmrs\") pod \"payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh\" (UID: \"6977f362-678e-4e26-bcc0-1e38c063fca5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh" Oct 30 19:07:43.652947 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:43.652908 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6977f362-678e-4e26-bcc0-1e38c063fca5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh\" (UID: \"6977f362-678e-4e26-bcc0-1e38c063fca5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh" Oct 30 19:07:43.653187 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:43.652967 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgmrs\" (UniqueName: \"kubernetes.io/projected/6977f362-678e-4e26-bcc0-1e38c063fca5-kube-api-access-fgmrs\") pod \"payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh\" (UID: \"6977f362-678e-4e26-bcc0-1e38c063fca5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh" Oct 30 19:07:43.653688 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:43.653667 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6977f362-678e-4e26-bcc0-1e38c063fca5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh\" (UID: \"6977f362-678e-4e26-bcc0-1e38c063fca5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh" Oct 30 19:07:43.714788 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:43.714755 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgmrs\" (UniqueName: \"kubernetes.io/projected/6977f362-678e-4e26-bcc0-1e38c063fca5-kube-api-access-fgmrs\") pod \"payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh\" (UID: \"6977f362-678e-4e26-bcc0-1e38c063fca5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh" Oct 30 19:07:43.809637 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:43.809523 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh" Oct 30 19:07:44.202028 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:44.201946 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx"] Oct 30 19:07:44.206771 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:44.206751 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx" Oct 30 19:07:44.222183 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:44.222154 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh"] Oct 30 19:07:44.223527 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:07:44.223501 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6977f362_678e_4e26_bcc0_1e38c063fca5.slice/crio-5a6a5fb9c3fd84093d67c1e58887a61493dcd55498ce5520dc3bbb95dc4c6f80 WatchSource:0}: Error finding container 5a6a5fb9c3fd84093d67c1e58887a61493dcd55498ce5520dc3bbb95dc4c6f80: Status 404 returned error can't find the container with id 5a6a5fb9c3fd84093d67c1e58887a61493dcd55498ce5520dc3bbb95dc4c6f80 Oct 30 19:07:44.265200 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:44.263550 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhhdt\" (UniqueName: \"kubernetes.io/projected/fefa5b0d-be5a-41f5-9f0c-ac483f457148-kube-api-access-jhhdt\") pod \"payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx\" (UID: \"fefa5b0d-be5a-41f5-9f0c-ac483f457148\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx" Oct 30 19:07:44.265200 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:44.263648 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fefa5b0d-be5a-41f5-9f0c-ac483f457148-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx\" (UID: \"fefa5b0d-be5a-41f5-9f0c-ac483f457148\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx" Oct 30 19:07:44.268915 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:44.268883 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx"] Oct 30 19:07:44.272236 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:44.272177 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh" event={"ID":"6977f362-678e-4e26-bcc0-1e38c063fca5","Type":"ContainerStarted","Data":"5a6a5fb9c3fd84093d67c1e58887a61493dcd55498ce5520dc3bbb95dc4c6f80"} Oct 30 19:07:44.274026 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:44.273999 4531 generic.go:356] "Generic (PLEG): container finished" podID="8f666e16-86ea-46f9-908f-48dcacfb331e" containerID="d0f8adc933bca5440734098d4d4a085dfd60916753b03a12dbe6b9898280f455" exitCode=0 Oct 30 19:07:44.274188 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:44.274069 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf" event={"ID":"8f666e16-86ea-46f9-908f-48dcacfb331e","Type":"ContainerDied","Data":"d0f8adc933bca5440734098d4d4a085dfd60916753b03a12dbe6b9898280f455"} Oct 30 19:07:44.364568 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:44.364537 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fefa5b0d-be5a-41f5-9f0c-ac483f457148-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx\" (UID: \"fefa5b0d-be5a-41f5-9f0c-ac483f457148\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx" Oct 30 19:07:44.364756 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:44.364679 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhhdt\" (UniqueName: \"kubernetes.io/projected/fefa5b0d-be5a-41f5-9f0c-ac483f457148-kube-api-access-jhhdt\") pod \"payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx\" (UID: \"fefa5b0d-be5a-41f5-9f0c-ac483f457148\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx" Oct 30 19:07:44.365274 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:44.365253 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fefa5b0d-be5a-41f5-9f0c-ac483f457148-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx\" (UID: \"fefa5b0d-be5a-41f5-9f0c-ac483f457148\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx" Oct 30 19:07:44.426251 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:44.426216 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhhdt\" (UniqueName: \"kubernetes.io/projected/fefa5b0d-be5a-41f5-9f0c-ac483f457148-kube-api-access-jhhdt\") pod \"payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx\" (UID: \"fefa5b0d-be5a-41f5-9f0c-ac483f457148\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx" Oct 30 19:07:44.520959 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:44.520928 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx" Oct 30 19:07:44.901109 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:07:44.901074 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefa5b0d_be5a_41f5_9f0c_ac483f457148.slice/crio-4ebc67ff875c702609f569e57b332f117d4e3d28397a9fc43526b6de1447aa15 WatchSource:0}: Error finding container 4ebc67ff875c702609f569e57b332f117d4e3d28397a9fc43526b6de1447aa15: Status 404 returned error can't find the container with id 4ebc67ff875c702609f569e57b332f117d4e3d28397a9fc43526b6de1447aa15 Oct 30 19:07:44.901827 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:44.901794 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx"] Oct 30 19:07:45.283039 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:45.282970 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh" event={"ID":"6977f362-678e-4e26-bcc0-1e38c063fca5","Type":"ContainerStarted","Data":"15b1d05288f5eeac71cd92e83866c369432d03977e44e4d622831dff0cc4de77"} Oct 30 19:07:45.283567 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:45.283211 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh" Oct 30 19:07:45.284848 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:45.284816 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx" event={"ID":"fefa5b0d-be5a-41f5-9f0c-ac483f457148","Type":"ContainerStarted","Data":"acfe7f76309ec7c235a21bc767ebe8003192114f480243b6390e7c5052896efc"} Oct 30 19:07:45.285012 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:45.284995 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx" Oct 30 19:07:45.285073 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:45.285034 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx" event={"ID":"fefa5b0d-be5a-41f5-9f0c-ac483f457148","Type":"ContainerStarted","Data":"4ebc67ff875c702609f569e57b332f117d4e3d28397a9fc43526b6de1447aa15"} Oct 30 19:07:45.402841 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:45.402146 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh" podStartSLOduration=2.402109891 podStartE2EDuration="2.402109891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:45.401468071 +0000 UTC m=+2025.428573031" watchObservedRunningTime="2026-10-30 19:07:45.402109891 +0000 UTC m=+2025.429214850" Oct 30 19:07:45.472731 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:45.472696 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf" Oct 30 19:07:45.543240 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:45.540239 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx" podStartSLOduration=1.54021538 podStartE2EDuration="1.54021538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:45.535779627 +0000 UTC m=+2025.562884587" watchObservedRunningTime="2026-10-30 19:07:45.54021538 +0000 UTC m=+2025.567320342" Oct 30 19:07:45.579671 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:45.579618 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f666e16-86ea-46f9-908f-48dcacfb331e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f666e16-86ea-46f9-908f-48dcacfb331e-maas-api-xks-ca\") pod \"8f666e16-86ea-46f9-908f-48dcacfb331e\" (UID: \"8f666e16-86ea-46f9-908f-48dcacfb331e\") " Oct 30 19:07:45.579837 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:45.579717 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f666e16-86ea-46f9-908f-48dcacfb331e-kube-api-access-2xscl\" (UniqueName: \"kubernetes.io/projected/8f666e16-86ea-46f9-908f-48dcacfb331e-kube-api-access-2xscl\") pod \"8f666e16-86ea-46f9-908f-48dcacfb331e\" (UID: \"8f666e16-86ea-46f9-908f-48dcacfb331e\") " Oct 30 19:07:45.579912 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:45.579895 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f666e16-86ea-46f9-908f-48dcacfb331e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f666e16-86ea-46f9-908f-48dcacfb331e-maas-api-openshift-ca\") pod \"8f666e16-86ea-46f9-908f-48dcacfb331e\" (UID: \"8f666e16-86ea-46f9-908f-48dcacfb331e\") " Oct 30 19:07:45.580403 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:45.580309 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f666e16-86ea-46f9-908f-48dcacfb331e-maas-api-openshift-ca" pod "8f666e16-86ea-46f9-908f-48dcacfb331e" (UID: "8f666e16-86ea-46f9-908f-48dcacfb331e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:07:45.580636 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:45.580509 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f666e16-86ea-46f9-908f-48dcacfb331e-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:07:45.582152 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:45.582102 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f666e16-86ea-46f9-908f-48dcacfb331e-maas-api-xks-ca" pod "8f666e16-86ea-46f9-908f-48dcacfb331e" (UID: "8f666e16-86ea-46f9-908f-48dcacfb331e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:07:45.583278 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:45.583250 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f666e16-86ea-46f9-908f-48dcacfb331e-kube-api-access-2xscl" pod "8f666e16-86ea-46f9-908f-48dcacfb331e" (UID: "8f666e16-86ea-46f9-908f-48dcacfb331e"). InnerVolumeSpecName "kube-api-access-2xscl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:07:45.683927 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:45.682587 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f666e16-86ea-46f9-908f-48dcacfb331e-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:07:45.683927 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:45.682625 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xscl\" (UniqueName: \"kubernetes.io/projected/8f666e16-86ea-46f9-908f-48dcacfb331e-kube-api-access-2xscl\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:07:46.229613 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:46.229576 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj"] Oct 30 19:07:46.229820 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:46.229792 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" podUID="2b5e06df-6296-4e17-8dfb-2c02e919e622" containerName="istio-proxy" containerID="cri-o://5d4a6f32308b9ca1319c373239bb4eb020806da1b1dd32ac70638ec873b428b2" gracePeriod=30 Oct 30 19:07:46.294961 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:46.294931 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf" Oct 30 19:07:46.295492 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:46.294962 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf" event={"ID":"8f666e16-86ea-46f9-908f-48dcacfb331e","Type":"ContainerDied","Data":"48710578b8208111a008c534f5eaf1b2e4f3a52d14ed205a2275381a0c5f6cd0"} Oct 30 19:07:46.295492 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:46.295016 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48710578b8208111a008c534f5eaf1b2e4f3a52d14ed205a2275381a0c5f6cd0" Oct 30 19:07:46.906305 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:46.906243 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn" podUID="6599d4b0-0958-4800-be76-df33b1e67a87" containerName="payload-processing" containerID="cri-o://ba34bcbe4abb90657f677d353b60a2a02d926883b235ea197a651db4f025eeed" gracePeriod=25 Oct 30 19:07:47.014586 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.014542 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx" podUID="4a126c43-8fec-4d89-b17e-9d34db70827f" containerName="payload-pre-processing" containerID="cri-o://a64978493f3f919f4e23c0745bd66927932be3c20b67367a45bfefa3d62d2f1f" gracePeriod=25 Oct 30 19:07:47.186222 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.186187 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn" Oct 30 19:07:47.301814 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.301783 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6599d4b0-0958-4800-be76-df33b1e67a87-kube-api-access-mhsfq\" (UniqueName: \"kubernetes.io/projected/6599d4b0-0958-4800-be76-df33b1e67a87-kube-api-access-mhsfq\") pod \"6599d4b0-0958-4800-be76-df33b1e67a87\" (UID: \"6599d4b0-0958-4800-be76-df33b1e67a87\") " Oct 30 19:07:47.302317 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.301900 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6599d4b0-0958-4800-be76-df33b1e67a87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6599d4b0-0958-4800-be76-df33b1e67a87-plugins-config-volume\") pod \"6599d4b0-0958-4800-be76-df33b1e67a87\" (UID: \"6599d4b0-0958-4800-be76-df33b1e67a87\") " Oct 30 19:07:47.302482 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.302454 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6599d4b0-0958-4800-be76-df33b1e67a87-plugins-config-volume" pod "6599d4b0-0958-4800-be76-df33b1e67a87" (UID: "6599d4b0-0958-4800-be76-df33b1e67a87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:07:47.302975 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.302951 4531 generic.go:356] "Generic (PLEG): container finished" podID="4a126c43-8fec-4d89-b17e-9d34db70827f" containerID="a64978493f3f919f4e23c0745bd66927932be3c20b67367a45bfefa3d62d2f1f" exitCode=0 Oct 30 19:07:47.303302 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.303280 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx" Oct 30 19:07:47.303420 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.303289 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx" event={"ID":"4a126c43-8fec-4d89-b17e-9d34db70827f","Type":"ContainerDied","Data":"a64978493f3f919f4e23c0745bd66927932be3c20b67367a45bfefa3d62d2f1f"} Oct 30 19:07:47.303470 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.303428 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx" event={"ID":"4a126c43-8fec-4d89-b17e-9d34db70827f","Type":"ContainerDied","Data":"502a3f4720bccc6ed764c68a6a9a476c79c9f9b5033b8adee5656522521af81d"} Oct 30 19:07:47.303470 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.303463 4531 scope.go:122] "RemoveContainer" containerID="a64978493f3f919f4e23c0745bd66927932be3c20b67367a45bfefa3d62d2f1f" Oct 30 19:07:47.305002 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.304976 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6599d4b0-0958-4800-be76-df33b1e67a87-kube-api-access-mhsfq" pod "6599d4b0-0958-4800-be76-df33b1e67a87" (UID: "6599d4b0-0958-4800-be76-df33b1e67a87"). InnerVolumeSpecName "kube-api-access-mhsfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:07:47.308965 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.308936 4531 generic.go:356] "Generic (PLEG): container finished" podID="6599d4b0-0958-4800-be76-df33b1e67a87" containerID="ba34bcbe4abb90657f677d353b60a2a02d926883b235ea197a651db4f025eeed" exitCode=0 Oct 30 19:07:47.309101 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.309003 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn" event={"ID":"6599d4b0-0958-4800-be76-df33b1e67a87","Type":"ContainerDied","Data":"ba34bcbe4abb90657f677d353b60a2a02d926883b235ea197a651db4f025eeed"} Oct 30 19:07:47.309101 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.309010 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn" Oct 30 19:07:47.309101 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.309052 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn" event={"ID":"6599d4b0-0958-4800-be76-df33b1e67a87","Type":"ContainerDied","Data":"1da77e5baa943340bc062527993ffe7178898710fa899412145458917f78a08c"} Oct 30 19:07:47.320880 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.320836 4531 scope.go:122] "RemoveContainer" containerID="a64978493f3f919f4e23c0745bd66927932be3c20b67367a45bfefa3d62d2f1f" Oct 30 19:07:47.321447 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:07:47.321332 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a64978493f3f919f4e23c0745bd66927932be3c20b67367a45bfefa3d62d2f1f\": container with ID starting with a64978493f3f919f4e23c0745bd66927932be3c20b67367a45bfefa3d62d2f1f not found: ID does not exist" containerID="a64978493f3f919f4e23c0745bd66927932be3c20b67367a45bfefa3d62d2f1f" Oct 30 19:07:47.321447 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.321394 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a64978493f3f919f4e23c0745bd66927932be3c20b67367a45bfefa3d62d2f1f"} err="failed to get container status \"a64978493f3f919f4e23c0745bd66927932be3c20b67367a45bfefa3d62d2f1f\": rpc error: code = NotFound desc = could not find container \"a64978493f3f919f4e23c0745bd66927932be3c20b67367a45bfefa3d62d2f1f\": container with ID starting with a64978493f3f919f4e23c0745bd66927932be3c20b67367a45bfefa3d62d2f1f not found: ID does not exist" Oct 30 19:07:47.321447 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.321420 4531 scope.go:122] "RemoveContainer" containerID="ba34bcbe4abb90657f677d353b60a2a02d926883b235ea197a651db4f025eeed" Oct 30 19:07:47.334338 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.334303 4531 scope.go:122] "RemoveContainer" containerID="ba34bcbe4abb90657f677d353b60a2a02d926883b235ea197a651db4f025eeed" Oct 30 19:07:47.334751 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:07:47.334724 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba34bcbe4abb90657f677d353b60a2a02d926883b235ea197a651db4f025eeed\": container with ID starting with ba34bcbe4abb90657f677d353b60a2a02d926883b235ea197a651db4f025eeed not found: ID does not exist" containerID="ba34bcbe4abb90657f677d353b60a2a02d926883b235ea197a651db4f025eeed" Oct 30 19:07:47.334880 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.334758 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba34bcbe4abb90657f677d353b60a2a02d926883b235ea197a651db4f025eeed"} err="failed to get container status \"ba34bcbe4abb90657f677d353b60a2a02d926883b235ea197a651db4f025eeed\": rpc error: code = NotFound desc = could not find container \"ba34bcbe4abb90657f677d353b60a2a02d926883b235ea197a651db4f025eeed\": container with ID starting with ba34bcbe4abb90657f677d353b60a2a02d926883b235ea197a651db4f025eeed not found: ID does not exist" Oct 30 19:07:47.403240 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.403204 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a126c43-8fec-4d89-b17e-9d34db70827f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a126c43-8fec-4d89-b17e-9d34db70827f-plugins-config-volume\") pod \"4a126c43-8fec-4d89-b17e-9d34db70827f\" (UID: \"4a126c43-8fec-4d89-b17e-9d34db70827f\") " Oct 30 19:07:47.403429 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.403298 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a126c43-8fec-4d89-b17e-9d34db70827f-kube-api-access-wljrl\" (UniqueName: \"kubernetes.io/projected/4a126c43-8fec-4d89-b17e-9d34db70827f-kube-api-access-wljrl\") pod \"4a126c43-8fec-4d89-b17e-9d34db70827f\" (UID: \"4a126c43-8fec-4d89-b17e-9d34db70827f\") " Oct 30 19:07:47.403618 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.403595 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a126c43-8fec-4d89-b17e-9d34db70827f-plugins-config-volume" pod "4a126c43-8fec-4d89-b17e-9d34db70827f" (UID: "4a126c43-8fec-4d89-b17e-9d34db70827f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:07:47.403618 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.403611 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhsfq\" (UniqueName: \"kubernetes.io/projected/6599d4b0-0958-4800-be76-df33b1e67a87-kube-api-access-mhsfq\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:07:47.403690 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.403624 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6599d4b0-0958-4800-be76-df33b1e67a87-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:07:47.405675 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.405636 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a126c43-8fec-4d89-b17e-9d34db70827f-kube-api-access-wljrl" pod "4a126c43-8fec-4d89-b17e-9d34db70827f" (UID: "4a126c43-8fec-4d89-b17e-9d34db70827f"). InnerVolumeSpecName "kube-api-access-wljrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:07:47.504416 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.504367 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a126c43-8fec-4d89-b17e-9d34db70827f-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:07:47.504416 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.504410 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wljrl\" (UniqueName: \"kubernetes.io/projected/4a126c43-8fec-4d89-b17e-9d34db70827f-kube-api-access-wljrl\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:07:47.538688 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.538647 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn"] Oct 30 19:07:47.538854 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:47.538705 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0ee379-6f47655788-6qzjn"] Oct 30 19:07:48.012898 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:48.012864 4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:07:48.013063 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:48.012929 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" podUID="2b5e06df-6296-4e17-8dfb-2c02e919e622" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:07:48.099015 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:48.098979 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6599d4b0-0958-4800-be76-df33b1e67a87" path="/var/lib/kubelet/pods/6599d4b0-0958-4800-be76-df33b1e67a87/volumes" Oct 30 19:07:48.317707 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:48.317670 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx" Oct 30 19:07:48.485505 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:48.485463 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx"] Oct 30 19:07:48.534840 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:48.534801 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-85bb59c4ff-dvcrx"] Oct 30 19:07:50.100076 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:50.100035 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a126c43-8fec-4d89-b17e-9d34db70827f" path="/var/lib/kubelet/pods/4a126c43-8fec-4d89-b17e-9d34db70827f/volumes" Oct 30 19:07:51.493755 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.493721 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 19:07:51.582088 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.581972 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-credential-socket\") pod \"2b5e06df-6296-4e17-8dfb-2c02e919e622\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " Oct 30 19:07:51.582088 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.582093 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-workload-socket\") pod \"2b5e06df-6296-4e17-8dfb-2c02e919e622\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " Oct 30 19:07:51.582403 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.582186 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-podinfo\") pod \"2b5e06df-6296-4e17-8dfb-2c02e919e622\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " Oct 30 19:07:51.582403 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.582212 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-workload-certs\") pod \"2b5e06df-6296-4e17-8dfb-2c02e919e622\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " Oct 30 19:07:51.582403 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.582257 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b5e06df-6296-4e17-8dfb-2c02e919e622-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5e06df-6296-4e17-8dfb-2c02e919e622-istiod-ca-cert\") pod \"2b5e06df-6296-4e17-8dfb-2c02e919e622\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " Oct 30 19:07:51.582403 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.582330 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-token\" (UniqueName: \"kubernetes.io/projected/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-token\") pod \"2b5e06df-6296-4e17-8dfb-2c02e919e622\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " Oct 30 19:07:51.582403 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.582357 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-credential-socket" pod "2b5e06df-6296-4e17-8dfb-2c02e919e622" (UID: "2b5e06df-6296-4e17-8dfb-2c02e919e622"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:07:51.582679 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.582411 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-envoy\") pod \"2b5e06df-6296-4e17-8dfb-2c02e919e622\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " Oct 30 19:07:51.582679 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.582456 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-data\") pod \"2b5e06df-6296-4e17-8dfb-2c02e919e622\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " Oct 30 19:07:51.582679 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.582528 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5e06df-6296-4e17-8dfb-2c02e919e622-kube-api-access-xtwp6\" (UniqueName: \"kubernetes.io/projected/2b5e06df-6296-4e17-8dfb-2c02e919e622-kube-api-access-xtwp6\") pod \"2b5e06df-6296-4e17-8dfb-2c02e919e622\" (UID: \"2b5e06df-6296-4e17-8dfb-2c02e919e622\") " Oct 30 19:07:51.582679 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.582534 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-workload-certs" pod "2b5e06df-6296-4e17-8dfb-2c02e919e622" (UID: "2b5e06df-6296-4e17-8dfb-2c02e919e622"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:07:51.582903 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.582741 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-workload-socket" pod "2b5e06df-6296-4e17-8dfb-2c02e919e622" (UID: "2b5e06df-6296-4e17-8dfb-2c02e919e622"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:07:51.582903 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.582758 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-data" pod "2b5e06df-6296-4e17-8dfb-2c02e919e622" (UID: "2b5e06df-6296-4e17-8dfb-2c02e919e622"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:07:51.582903 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.582764 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b5e06df-6296-4e17-8dfb-2c02e919e622-istiod-ca-cert" pod "2b5e06df-6296-4e17-8dfb-2c02e919e622" (UID: "2b5e06df-6296-4e17-8dfb-2c02e919e622"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:07:51.583284 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.583255 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:07:51.583390 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.583285 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:07:51.583390 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.583300 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:07:51.583390 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.583315 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:07:51.583390 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.583330 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5e06df-6296-4e17-8dfb-2c02e919e622-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:07:51.584724 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.584698 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-podinfo" pod "2b5e06df-6296-4e17-8dfb-2c02e919e622" (UID: "2b5e06df-6296-4e17-8dfb-2c02e919e622"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:07:51.584809 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.584757 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-token" pod "2b5e06df-6296-4e17-8dfb-2c02e919e622" (UID: "2b5e06df-6296-4e17-8dfb-2c02e919e622"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:07:51.584865 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.584834 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-envoy" pod "2b5e06df-6296-4e17-8dfb-2c02e919e622" (UID: "2b5e06df-6296-4e17-8dfb-2c02e919e622"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:07:51.585292 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.585266 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5e06df-6296-4e17-8dfb-2c02e919e622-kube-api-access-xtwp6" pod "2b5e06df-6296-4e17-8dfb-2c02e919e622" (UID: "2b5e06df-6296-4e17-8dfb-2c02e919e622"). InnerVolumeSpecName "kube-api-access-xtwp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:07:51.684638 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.684603 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:07:51.684638 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.684630 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:07:51.684638 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.684641 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5e06df-6296-4e17-8dfb-2c02e919e622-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:07:51.684870 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:51.684656 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtwp6\" (UniqueName: \"kubernetes.io/projected/2b5e06df-6296-4e17-8dfb-2c02e919e622-kube-api-access-xtwp6\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:07:52.344712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:52.344675 4531 generic.go:356] "Generic (PLEG): container finished" podID="2b5e06df-6296-4e17-8dfb-2c02e919e622" containerID="5d4a6f32308b9ca1319c373239bb4eb020806da1b1dd32ac70638ec873b428b2" exitCode=0 Oct 30 19:07:52.344883 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:52.344722 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" event={"ID":"2b5e06df-6296-4e17-8dfb-2c02e919e622","Type":"ContainerDied","Data":"5d4a6f32308b9ca1319c373239bb4eb020806da1b1dd32ac70638ec873b428b2"} Oct 30 19:07:52.344883 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:52.344786 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" event={"ID":"2b5e06df-6296-4e17-8dfb-2c02e919e622","Type":"ContainerDied","Data":"04e49c2a812dca714a8c47a4321e2cf40a65bb7783fd3b0a16d91fb0a5f3c2c8"} Oct 30 19:07:52.344883 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:52.344813 4531 scope.go:122] "RemoveContainer" containerID="5d4a6f32308b9ca1319c373239bb4eb020806da1b1dd32ac70638ec873b428b2" Oct 30 19:07:52.344883 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:52.344733 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj" Oct 30 19:07:52.355252 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:52.355227 4531 scope.go:122] "RemoveContainer" containerID="5d4a6f32308b9ca1319c373239bb4eb020806da1b1dd32ac70638ec873b428b2" Oct 30 19:07:52.355650 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:07:52.355630 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d4a6f32308b9ca1319c373239bb4eb020806da1b1dd32ac70638ec873b428b2\": container with ID starting with 5d4a6f32308b9ca1319c373239bb4eb020806da1b1dd32ac70638ec873b428b2 not found: ID does not exist" containerID="5d4a6f32308b9ca1319c373239bb4eb020806da1b1dd32ac70638ec873b428b2" Oct 30 19:07:52.355707 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:52.355659 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d4a6f32308b9ca1319c373239bb4eb020806da1b1dd32ac70638ec873b428b2"} err="failed to get container status \"5d4a6f32308b9ca1319c373239bb4eb020806da1b1dd32ac70638ec873b428b2\": rpc error: code = NotFound desc = could not find container \"5d4a6f32308b9ca1319c373239bb4eb020806da1b1dd32ac70638ec873b428b2\": container with ID starting with 5d4a6f32308b9ca1319c373239bb4eb020806da1b1dd32ac70638ec873b428b2 not found: ID does not exist" Oct 30 19:07:52.559019 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:52.558980 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj"] Oct 30 19:07:52.608425 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:52.608378 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-99swj"] Oct 30 19:07:54.098431 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:54.098389 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b5e06df-6296-4e17-8dfb-2c02e919e622" path="/var/lib/kubelet/pods/2b5e06df-6296-4e17-8dfb-2c02e919e622/volumes" Oct 30 19:07:56.298036 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:56.298006 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh" Oct 30 19:07:56.298462 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:07:56.298085 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx" Oct 30 19:07:56.399167 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:07:56.399096 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:08:01.539707 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:01.539671 4531 scope.go:122] "RemoveContainer" containerID="d6106021d164c92114c1044ab15c9f99887e3e32747ab0d5fcba5b4b53f3753e" Oct 30 19:08:01.549229 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:01.549208 4531 scope.go:122] "RemoveContainer" containerID="2cc141d768e360a0035e711ebbdc4d029da586fbab0657cc9aa24e21d262bb19" Oct 30 19:08:01.560942 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:01.560917 4531 scope.go:122] "RemoveContainer" containerID="19a2977eaa3897cc33c02c364d63cd2ded5ce45f10074aef1bce02c2213c18bd" Oct 30 19:08:08.830461 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:08.830368 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4"] Oct 30 19:08:08.831778 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:08.831732 4531 cpu_manager.go:414] "containerMap: removing container" podUID="6599d4b0-0958-4800-be76-df33b1e67a87" containerName="payload-processing" Oct 30 19:08:08.831954 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:08.831937 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6599d4b0-0958-4800-be76-df33b1e67a87" containerName="payload-processing" Oct 30 19:08:08.832064 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:08.832051 4531 cpu_manager.go:414] "containerMap: removing container" podUID="8f666e16-86ea-46f9-908f-48dcacfb331e" containerName="revoke-keys" Oct 30 19:08:08.832189 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:08.832175 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f666e16-86ea-46f9-908f-48dcacfb331e" containerName="revoke-keys" Oct 30 19:08:08.832303 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:08.832292 4531 cpu_manager.go:414] "containerMap: removing container" podUID="2b5e06df-6296-4e17-8dfb-2c02e919e622" containerName="istio-proxy" Oct 30 19:08:08.832378 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:08.832369 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b5e06df-6296-4e17-8dfb-2c02e919e622" containerName="istio-proxy" Oct 30 19:08:08.832481 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:08.832469 4531 cpu_manager.go:414] "containerMap: removing container" podUID="4a126c43-8fec-4d89-b17e-9d34db70827f" containerName="payload-pre-processing" Oct 30 19:08:08.832567 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:08.832555 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a126c43-8fec-4d89-b17e-9d34db70827f" containerName="payload-pre-processing" Oct 30 19:08:08.832801 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:08.832784 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a126c43-8fec-4d89-b17e-9d34db70827f" containerName="payload-pre-processing" Oct 30 19:08:08.832919 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:08.832908 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6599d4b0-0958-4800-be76-df33b1e67a87" containerName="payload-processing" Oct 30 19:08:08.832997 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:08.832988 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f666e16-86ea-46f9-908f-48dcacfb331e" containerName="revoke-keys" Oct 30 19:08:08.833116 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:08.833103 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b5e06df-6296-4e17-8dfb-2c02e919e622" containerName="istio-proxy" Oct 30 19:08:08.846371 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:08.839218 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4" Oct 30 19:08:08.898166 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:08.898105 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4"] Oct 30 19:08:08.911219 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:08.911181 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0df638c-bae0-471f-94a8-60e0fe66bbb5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4\" (UID: \"f0df638c-bae0-471f-94a8-60e0fe66bbb5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4" Oct 30 19:08:08.911409 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:08.911349 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r84sh\" (UniqueName: \"kubernetes.io/projected/f0df638c-bae0-471f-94a8-60e0fe66bbb5-kube-api-access-r84sh\") pod \"payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4\" (UID: \"f0df638c-bae0-471f-94a8-60e0fe66bbb5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4" Oct 30 19:08:08.943679 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:08.943636 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4ebf78\"" Oct 30 19:08:08.943881 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:08.943700 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4ebf78-dockercfg-jnpcr\"" Oct 30 19:08:08.997855 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:08.997819 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w"] Oct 30 19:08:09.001743 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:09.001720 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w" Oct 30 19:08:09.012454 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:09.012420 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0df638c-bae0-471f-94a8-60e0fe66bbb5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4\" (UID: \"f0df638c-bae0-471f-94a8-60e0fe66bbb5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4" Oct 30 19:08:09.012591 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:09.012564 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r84sh\" (UniqueName: \"kubernetes.io/projected/f0df638c-bae0-471f-94a8-60e0fe66bbb5-kube-api-access-r84sh\") pod \"payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4\" (UID: \"f0df638c-bae0-471f-94a8-60e0fe66bbb5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4" Oct 30 19:08:09.013088 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:09.013068 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0df638c-bae0-471f-94a8-60e0fe66bbb5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4\" (UID: \"f0df638c-bae0-471f-94a8-60e0fe66bbb5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4" Oct 30 19:08:09.062807 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:09.062768 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w"] Oct 30 19:08:09.073889 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:09.073862 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r84sh\" (UniqueName: \"kubernetes.io/projected/f0df638c-bae0-471f-94a8-60e0fe66bbb5-kube-api-access-r84sh\") pod \"payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4\" (UID: \"f0df638c-bae0-471f-94a8-60e0fe66bbb5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4" Oct 30 19:08:09.114156 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:09.114044 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76f80da7-3a86-4d0e-ba0b-167f66abeeba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w\" (UID: \"76f80da7-3a86-4d0e-ba0b-167f66abeeba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w" Oct 30 19:08:09.114320 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:09.114197 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvckz\" (UniqueName: \"kubernetes.io/projected/76f80da7-3a86-4d0e-ba0b-167f66abeeba-kube-api-access-wvckz\") pod \"payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w\" (UID: \"76f80da7-3a86-4d0e-ba0b-167f66abeeba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w" Oct 30 19:08:09.168780 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:09.168744 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4" Oct 30 19:08:09.215688 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:09.215644 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76f80da7-3a86-4d0e-ba0b-167f66abeeba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w\" (UID: \"76f80da7-3a86-4d0e-ba0b-167f66abeeba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w" Oct 30 19:08:09.215880 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:09.215748 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvckz\" (UniqueName: \"kubernetes.io/projected/76f80da7-3a86-4d0e-ba0b-167f66abeeba-kube-api-access-wvckz\") pod \"payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w\" (UID: \"76f80da7-3a86-4d0e-ba0b-167f66abeeba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w" Oct 30 19:08:09.216419 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:09.216392 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76f80da7-3a86-4d0e-ba0b-167f66abeeba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w\" (UID: \"76f80da7-3a86-4d0e-ba0b-167f66abeeba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w" Oct 30 19:08:09.277693 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:09.277657 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvckz\" (UniqueName: \"kubernetes.io/projected/76f80da7-3a86-4d0e-ba0b-167f66abeeba-kube-api-access-wvckz\") pod \"payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w\" (UID: \"76f80da7-3a86-4d0e-ba0b-167f66abeeba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w" Oct 30 19:08:09.315843 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:09.315575 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w" Oct 30 19:08:09.589646 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:09.589618 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4"] Oct 30 19:08:09.592368 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:08:09.592333 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0df638c_bae0_471f_94a8_60e0fe66bbb5.slice/crio-f019584ba7240b9db7fbb8fec32d14d5af9f8697eef45b21b73f4b68bf131245 WatchSource:0}: Error finding container f019584ba7240b9db7fbb8fec32d14d5af9f8697eef45b21b73f4b68bf131245: Status 404 returned error can't find the container with id f019584ba7240b9db7fbb8fec32d14d5af9f8697eef45b21b73f4b68bf131245 Oct 30 19:08:09.705479 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:09.704809 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w"] Oct 30 19:08:09.726770 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:08:09.726729 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f80da7_3a86_4d0e_ba0b_167f66abeeba.slice/crio-ca3b8cfab705942909ad1cf8ef69ad190f525b0c5c06be8b36ca0d5c05a6e224 WatchSource:0}: Error finding container ca3b8cfab705942909ad1cf8ef69ad190f525b0c5c06be8b36ca0d5c05a6e224: Status 404 returned error can't find the container with id ca3b8cfab705942909ad1cf8ef69ad190f525b0c5c06be8b36ca0d5c05a6e224 Oct 30 19:08:09.753974 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:09.753936 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf"] Oct 30 19:08:09.754293 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:09.754252 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf" podUID="72725b3c-582d-44a2-9ca7-69fa4cfe2982" containerName="payload-processing" containerID="cri-o://f903f726cbc900b0972772e1da6d1e037e928df3b009e46fe7d6542a02d7b40b" gracePeriod=30 Oct 30 19:08:09.810957 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:09.810924 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx"] Oct 30 19:08:09.811180 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:09.811120 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx" podUID="406b8a13-31e1-4660-a01a-c482d61f8fd1" containerName="payload-pre-processing" containerID="cri-o://d7dca23bb80fb466735fe9bd8e0d825683193606def9dd0d91451e50782eab58" gracePeriod=30 Oct 30 19:08:10.072442 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.072418 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf" Oct 30 19:08:10.113428 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.113397 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx" Oct 30 19:08:10.128176 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.128115 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72725b3c-582d-44a2-9ca7-69fa4cfe2982-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72725b3c-582d-44a2-9ca7-69fa4cfe2982-plugins-config-volume\") pod \"72725b3c-582d-44a2-9ca7-69fa4cfe2982\" (UID: \"72725b3c-582d-44a2-9ca7-69fa4cfe2982\") " Oct 30 19:08:10.128385 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.128235 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72725b3c-582d-44a2-9ca7-69fa4cfe2982-kube-api-access-749mw\" (UniqueName: \"kubernetes.io/projected/72725b3c-582d-44a2-9ca7-69fa4cfe2982-kube-api-access-749mw\") pod \"72725b3c-582d-44a2-9ca7-69fa4cfe2982\" (UID: \"72725b3c-582d-44a2-9ca7-69fa4cfe2982\") " Oct 30 19:08:10.128526 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.128500 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72725b3c-582d-44a2-9ca7-69fa4cfe2982-plugins-config-volume" pod "72725b3c-582d-44a2-9ca7-69fa4cfe2982" (UID: "72725b3c-582d-44a2-9ca7-69fa4cfe2982"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:10.128653 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.128639 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72725b3c-582d-44a2-9ca7-69fa4cfe2982-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:10.130585 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.130557 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72725b3c-582d-44a2-9ca7-69fa4cfe2982-kube-api-access-749mw" pod "72725b3c-582d-44a2-9ca7-69fa4cfe2982" (UID: "72725b3c-582d-44a2-9ca7-69fa4cfe2982"). InnerVolumeSpecName "kube-api-access-749mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:10.230049 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.229954 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/406b8a13-31e1-4660-a01a-c482d61f8fd1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/406b8a13-31e1-4660-a01a-c482d61f8fd1-plugins-config-volume\") pod \"406b8a13-31e1-4660-a01a-c482d61f8fd1\" (UID: \"406b8a13-31e1-4660-a01a-c482d61f8fd1\") " Oct 30 19:08:10.230236 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.230049 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/406b8a13-31e1-4660-a01a-c482d61f8fd1-kube-api-access-rsnqx\" (UniqueName: \"kubernetes.io/projected/406b8a13-31e1-4660-a01a-c482d61f8fd1-kube-api-access-rsnqx\") pod \"406b8a13-31e1-4660-a01a-c482d61f8fd1\" (UID: \"406b8a13-31e1-4660-a01a-c482d61f8fd1\") " Oct 30 19:08:10.230462 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.230425 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/406b8a13-31e1-4660-a01a-c482d61f8fd1-plugins-config-volume" pod "406b8a13-31e1-4660-a01a-c482d61f8fd1" (UID: "406b8a13-31e1-4660-a01a-c482d61f8fd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:10.230669 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.230606 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/406b8a13-31e1-4660-a01a-c482d61f8fd1-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:10.230669 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.230626 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-749mw\" (UniqueName: \"kubernetes.io/projected/72725b3c-582d-44a2-9ca7-69fa4cfe2982-kube-api-access-749mw\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:10.232550 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.232516 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/406b8a13-31e1-4660-a01a-c482d61f8fd1-kube-api-access-rsnqx" pod "406b8a13-31e1-4660-a01a-c482d61f8fd1" (UID: "406b8a13-31e1-4660-a01a-c482d61f8fd1"). InnerVolumeSpecName "kube-api-access-rsnqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:10.334603 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.334511 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsnqx\" (UniqueName: \"kubernetes.io/projected/406b8a13-31e1-4660-a01a-c482d61f8fd1-kube-api-access-rsnqx\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:10.455102 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.455065 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w" event={"ID":"76f80da7-3a86-4d0e-ba0b-167f66abeeba","Type":"ContainerStarted","Data":"1321e66ff3bacf96ecda5fcb40136a8ccbe437e39c75ad861e7cb6164f915832"} Oct 30 19:08:10.455290 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.455140 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w" Oct 30 19:08:10.455290 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.455161 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w" event={"ID":"76f80da7-3a86-4d0e-ba0b-167f66abeeba","Type":"ContainerStarted","Data":"ca3b8cfab705942909ad1cf8ef69ad190f525b0c5c06be8b36ca0d5c05a6e224"} Oct 30 19:08:10.456442 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.456416 4531 generic.go:356] "Generic (PLEG): container finished" podID="406b8a13-31e1-4660-a01a-c482d61f8fd1" containerID="d7dca23bb80fb466735fe9bd8e0d825683193606def9dd0d91451e50782eab58" exitCode=0 Oct 30 19:08:10.456540 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.456453 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx" Oct 30 19:08:10.456540 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.456502 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx" event={"ID":"406b8a13-31e1-4660-a01a-c482d61f8fd1","Type":"ContainerDied","Data":"d7dca23bb80fb466735fe9bd8e0d825683193606def9dd0d91451e50782eab58"} Oct 30 19:08:10.456673 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.456545 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx" event={"ID":"406b8a13-31e1-4660-a01a-c482d61f8fd1","Type":"ContainerDied","Data":"1b76714af5bf9d754f8519b9fac526e82600b75076865e2953b9d7bb136dc319"} Oct 30 19:08:10.456673 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.456571 4531 scope.go:122] "RemoveContainer" containerID="d7dca23bb80fb466735fe9bd8e0d825683193606def9dd0d91451e50782eab58" Oct 30 19:08:10.457954 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.457931 4531 generic.go:356] "Generic (PLEG): container finished" podID="72725b3c-582d-44a2-9ca7-69fa4cfe2982" containerID="f903f726cbc900b0972772e1da6d1e037e928df3b009e46fe7d6542a02d7b40b" exitCode=0 Oct 30 19:08:10.458151 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.457960 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf" event={"ID":"72725b3c-582d-44a2-9ca7-69fa4cfe2982","Type":"ContainerDied","Data":"f903f726cbc900b0972772e1da6d1e037e928df3b009e46fe7d6542a02d7b40b"} Oct 30 19:08:10.458151 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.458008 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf" event={"ID":"72725b3c-582d-44a2-9ca7-69fa4cfe2982","Type":"ContainerDied","Data":"1345b59bfcf420934cc37cbb4ea47dfdfdc2c041cfaaa87b60f394f069b1863b"} Oct 30 19:08:10.458151 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.458040 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf" Oct 30 19:08:10.459594 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.459574 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4" event={"ID":"f0df638c-bae0-471f-94a8-60e0fe66bbb5","Type":"ContainerStarted","Data":"202c25f37e6421dd9475f6520a6a910eb0085b432b5d8d0f615f22b6e2f11693"} Oct 30 19:08:10.459702 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.459608 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4" event={"ID":"f0df638c-bae0-471f-94a8-60e0fe66bbb5","Type":"ContainerStarted","Data":"f019584ba7240b9db7fbb8fec32d14d5af9f8697eef45b21b73f4b68bf131245"} Oct 30 19:08:10.459702 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.459641 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4" Oct 30 19:08:10.468299 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.468272 4531 scope.go:122] "RemoveContainer" containerID="d7dca23bb80fb466735fe9bd8e0d825683193606def9dd0d91451e50782eab58" Oct 30 19:08:10.468622 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:08:10.468601 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7dca23bb80fb466735fe9bd8e0d825683193606def9dd0d91451e50782eab58\": container with ID starting with d7dca23bb80fb466735fe9bd8e0d825683193606def9dd0d91451e50782eab58 not found: ID does not exist" containerID="d7dca23bb80fb466735fe9bd8e0d825683193606def9dd0d91451e50782eab58" Oct 30 19:08:10.468721 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.468632 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7dca23bb80fb466735fe9bd8e0d825683193606def9dd0d91451e50782eab58"} err="failed to get container status \"d7dca23bb80fb466735fe9bd8e0d825683193606def9dd0d91451e50782eab58\": rpc error: code = NotFound desc = could not find container \"d7dca23bb80fb466735fe9bd8e0d825683193606def9dd0d91451e50782eab58\": container with ID starting with d7dca23bb80fb466735fe9bd8e0d825683193606def9dd0d91451e50782eab58 not found: ID does not exist" Oct 30 19:08:10.468721 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.468656 4531 scope.go:122] "RemoveContainer" containerID="f903f726cbc900b0972772e1da6d1e037e928df3b009e46fe7d6542a02d7b40b" Oct 30 19:08:10.477312 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.477295 4531 scope.go:122] "RemoveContainer" containerID="f903f726cbc900b0972772e1da6d1e037e928df3b009e46fe7d6542a02d7b40b" Oct 30 19:08:10.477581 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:08:10.477563 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f903f726cbc900b0972772e1da6d1e037e928df3b009e46fe7d6542a02d7b40b\": container with ID starting with f903f726cbc900b0972772e1da6d1e037e928df3b009e46fe7d6542a02d7b40b not found: ID does not exist" containerID="f903f726cbc900b0972772e1da6d1e037e928df3b009e46fe7d6542a02d7b40b" Oct 30 19:08:10.477647 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.477588 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f903f726cbc900b0972772e1da6d1e037e928df3b009e46fe7d6542a02d7b40b"} err="failed to get container status \"f903f726cbc900b0972772e1da6d1e037e928df3b009e46fe7d6542a02d7b40b\": rpc error: code = NotFound desc = could not find container \"f903f726cbc900b0972772e1da6d1e037e928df3b009e46fe7d6542a02d7b40b\": container with ID starting with f903f726cbc900b0972772e1da6d1e037e928df3b009e46fe7d6542a02d7b40b not found: ID does not exist" Oct 30 19:08:10.572393 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.572342 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w" podStartSLOduration=2.572328418 podStartE2EDuration="2.572328418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:10.570752674 +0000 UTC m=+2050.597857633" watchObservedRunningTime="2026-10-30 19:08:10.572328418 +0000 UTC m=+2050.599433417" Oct 30 19:08:10.687793 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.687721 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx"] Oct 30 19:08:10.694906 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.694866 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a23af5-64696c9fc9-t6nrx"] Oct 30 19:08:10.808520 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.808464 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4" podStartSLOduration=2.808448617 podStartE2EDuration="2.808448617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:10.806764847 +0000 UTC m=+2050.833869808" watchObservedRunningTime="2026-10-30 19:08:10.808448617 +0000 UTC m=+2050.835553575" Oct 30 19:08:10.918983 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.918875 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf"] Oct 30 19:08:10.926000 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:10.925113 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a23af5-5b87d549d6-m44zf"] Oct 30 19:08:11.484909 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:11.484861 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf"] Oct 30 19:08:11.530778 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:11.530732 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a23af5-n28wf"] Oct 30 19:08:12.098275 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:12.098243 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="406b8a13-31e1-4660-a01a-c482d61f8fd1" path="/var/lib/kubelet/pods/406b8a13-31e1-4660-a01a-c482d61f8fd1/volumes" Oct 30 19:08:12.098670 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:12.098657 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72725b3c-582d-44a2-9ca7-69fa4cfe2982" path="/var/lib/kubelet/pods/72725b3c-582d-44a2-9ca7-69fa4cfe2982/volumes" Oct 30 19:08:12.099036 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:12.099023 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f666e16-86ea-46f9-908f-48dcacfb331e" path="/var/lib/kubelet/pods/8f666e16-86ea-46f9-908f-48dcacfb331e/volumes" Oct 30 19:08:19.774019 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:19.773979 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz"] Oct 30 19:08:19.774593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:19.774576 4531 cpu_manager.go:414] "containerMap: removing container" podUID="406b8a13-31e1-4660-a01a-c482d61f8fd1" containerName="payload-pre-processing" Oct 30 19:08:19.774641 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:19.774597 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="406b8a13-31e1-4660-a01a-c482d61f8fd1" containerName="payload-pre-processing" Oct 30 19:08:19.774641 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:19.774625 4531 cpu_manager.go:414] "containerMap: removing container" podUID="72725b3c-582d-44a2-9ca7-69fa4cfe2982" containerName="payload-processing" Oct 30 19:08:19.774641 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:19.774630 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72725b3c-582d-44a2-9ca7-69fa4cfe2982" containerName="payload-processing" Oct 30 19:08:19.774741 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:19.774732 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="406b8a13-31e1-4660-a01a-c482d61f8fd1" containerName="payload-pre-processing" Oct 30 19:08:19.774778 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:19.774744 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="72725b3c-582d-44a2-9ca7-69fa4cfe2982" containerName="payload-processing" Oct 30 19:08:19.779700 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:19.779672 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz" Oct 30 19:08:19.832858 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:19.832815 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:08:19.860866 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:19.860822 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540058b9-e9db-4405-9dd1-03a4f85e50d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz\" (UID: \"540058b9-e9db-4405-9dd1-03a4f85e50d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz" Oct 30 19:08:19.861084 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:19.861057 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540058b9-e9db-4405-9dd1-03a4f85e50d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz\" (UID: \"540058b9-e9db-4405-9dd1-03a4f85e50d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz" Oct 30 19:08:19.861161 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:19.861113 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhwm5\" (UniqueName: \"kubernetes.io/projected/540058b9-e9db-4405-9dd1-03a4f85e50d6-kube-api-access-lhwm5\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz\" (UID: \"540058b9-e9db-4405-9dd1-03a4f85e50d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz" Oct 30 19:08:19.882255 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:19.882222 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:08:19.882474 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:19.882298 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:08:19.882540 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:19.882476 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:08:19.884990 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:19.884963 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz"] Oct 30 19:08:19.962082 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:19.962024 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540058b9-e9db-4405-9dd1-03a4f85e50d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz\" (UID: \"540058b9-e9db-4405-9dd1-03a4f85e50d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz" Oct 30 19:08:19.962324 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:19.962175 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540058b9-e9db-4405-9dd1-03a4f85e50d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz\" (UID: \"540058b9-e9db-4405-9dd1-03a4f85e50d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz" Oct 30 19:08:19.962324 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:19.962200 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhwm5\" (UniqueName: \"kubernetes.io/projected/540058b9-e9db-4405-9dd1-03a4f85e50d6-kube-api-access-lhwm5\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz\" (UID: \"540058b9-e9db-4405-9dd1-03a4f85e50d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz" Oct 30 19:08:19.962734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:19.962712 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540058b9-e9db-4405-9dd1-03a4f85e50d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz\" (UID: \"540058b9-e9db-4405-9dd1-03a4f85e50d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz" Oct 30 19:08:19.964837 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:19.964811 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540058b9-e9db-4405-9dd1-03a4f85e50d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz\" (UID: \"540058b9-e9db-4405-9dd1-03a4f85e50d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz" Oct 30 19:08:20.025327 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:20.025240 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhwm5\" (UniqueName: \"kubernetes.io/projected/540058b9-e9db-4405-9dd1-03a4f85e50d6-kube-api-access-lhwm5\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz\" (UID: \"540058b9-e9db-4405-9dd1-03a4f85e50d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz" Oct 30 19:08:20.093894 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:20.093853 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz" Oct 30 19:08:20.488421 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:20.488391 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz"] Oct 30 19:08:20.490715 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:08:20.490681 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540058b9_e9db_4405_9dd1_03a4f85e50d6.slice/crio-e9bf2ce68fc78cd7dbfacdf2c4a0bae13f21c53bad66df27eb80e6a802c68dbb WatchSource:0}: Error finding container e9bf2ce68fc78cd7dbfacdf2c4a0bae13f21c53bad66df27eb80e6a802c68dbb: Status 404 returned error can't find the container with id e9bf2ce68fc78cd7dbfacdf2c4a0bae13f21c53bad66df27eb80e6a802c68dbb Oct 30 19:08:20.522727 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:20.522695 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz" event={"ID":"540058b9-e9db-4405-9dd1-03a4f85e50d6","Type":"ContainerStarted","Data":"e9bf2ce68fc78cd7dbfacdf2c4a0bae13f21c53bad66df27eb80e6a802c68dbb"} Oct 30 19:08:21.471824 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:21.471787 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4" Oct 30 19:08:21.472331 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:21.472184 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w" Oct 30 19:08:21.529970 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:21.529932 4531 generic.go:356] "Generic (PLEG): container finished" podID="540058b9-e9db-4405-9dd1-03a4f85e50d6" containerID="186f0e704a2215a78cb0d43ab8f6120cd722fa8183e5305fd1441d4457d4ca7b" exitCode=0 Oct 30 19:08:21.530175 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:21.530027 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz" event={"ID":"540058b9-e9db-4405-9dd1-03a4f85e50d6","Type":"ContainerDied","Data":"186f0e704a2215a78cb0d43ab8f6120cd722fa8183e5305fd1441d4457d4ca7b"} Oct 30 19:08:22.349663 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:22.349626 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl"] Oct 30 19:08:22.353617 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:22.353596 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl" Oct 30 19:08:22.406008 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:22.405979 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0c9a2b\"" Oct 30 19:08:22.406008 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:22.405981 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0c9a2b-dockercfg-hzdfc\"" Oct 30 19:08:22.458533 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:22.458488 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl"] Oct 30 19:08:22.495915 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:22.495869 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/194c0736-90dd-41b0-b316-9858113b2477-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl\" (UID: \"194c0736-90dd-41b0-b316-9858113b2477\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl" Oct 30 19:08:22.496380 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:22.495931 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jssqx\" (UniqueName: \"kubernetes.io/projected/194c0736-90dd-41b0-b316-9858113b2477-kube-api-access-jssqx\") pod \"payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl\" (UID: \"194c0736-90dd-41b0-b316-9858113b2477\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl" Oct 30 19:08:22.596914 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:22.596872 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/194c0736-90dd-41b0-b316-9858113b2477-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl\" (UID: \"194c0736-90dd-41b0-b316-9858113b2477\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl" Oct 30 19:08:22.597123 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:22.596951 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jssqx\" (UniqueName: \"kubernetes.io/projected/194c0736-90dd-41b0-b316-9858113b2477-kube-api-access-jssqx\") pod \"payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl\" (UID: \"194c0736-90dd-41b0-b316-9858113b2477\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl" Oct 30 19:08:22.597849 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:22.597810 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/194c0736-90dd-41b0-b316-9858113b2477-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl\" (UID: \"194c0736-90dd-41b0-b316-9858113b2477\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl" Oct 30 19:08:22.659004 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:22.658925 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jssqx\" (UniqueName: \"kubernetes.io/projected/194c0736-90dd-41b0-b316-9858113b2477-kube-api-access-jssqx\") pod \"payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl\" (UID: \"194c0736-90dd-41b0-b316-9858113b2477\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl" Oct 30 19:08:22.674591 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:22.674551 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl" Oct 30 19:08:22.712352 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:22.712031 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz" Oct 30 19:08:22.799728 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:22.799681 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/540058b9-e9db-4405-9dd1-03a4f85e50d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540058b9-e9db-4405-9dd1-03a4f85e50d6-maas-api-xks-ca\") pod \"540058b9-e9db-4405-9dd1-03a4f85e50d6\" (UID: \"540058b9-e9db-4405-9dd1-03a4f85e50d6\") " Oct 30 19:08:22.799944 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:22.799741 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/540058b9-e9db-4405-9dd1-03a4f85e50d6-kube-api-access-lhwm5\" (UniqueName: \"kubernetes.io/projected/540058b9-e9db-4405-9dd1-03a4f85e50d6-kube-api-access-lhwm5\") pod \"540058b9-e9db-4405-9dd1-03a4f85e50d6\" (UID: \"540058b9-e9db-4405-9dd1-03a4f85e50d6\") " Oct 30 19:08:22.799944 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:22.799798 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/540058b9-e9db-4405-9dd1-03a4f85e50d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540058b9-e9db-4405-9dd1-03a4f85e50d6-maas-api-openshift-ca\") pod \"540058b9-e9db-4405-9dd1-03a4f85e50d6\" (UID: \"540058b9-e9db-4405-9dd1-03a4f85e50d6\") " Oct 30 19:08:22.800222 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:22.800197 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/540058b9-e9db-4405-9dd1-03a4f85e50d6-maas-api-openshift-ca" pod "540058b9-e9db-4405-9dd1-03a4f85e50d6" (UID: "540058b9-e9db-4405-9dd1-03a4f85e50d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:22.800390 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:22.800367 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540058b9-e9db-4405-9dd1-03a4f85e50d6-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:22.802430 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:22.802397 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/540058b9-e9db-4405-9dd1-03a4f85e50d6-maas-api-xks-ca" pod "540058b9-e9db-4405-9dd1-03a4f85e50d6" (UID: "540058b9-e9db-4405-9dd1-03a4f85e50d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:08:22.802589 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:22.802441 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/540058b9-e9db-4405-9dd1-03a4f85e50d6-kube-api-access-lhwm5" pod "540058b9-e9db-4405-9dd1-03a4f85e50d6" (UID: "540058b9-e9db-4405-9dd1-03a4f85e50d6"). InnerVolumeSpecName "kube-api-access-lhwm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:22.901998 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:22.901951 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540058b9-e9db-4405-9dd1-03a4f85e50d6-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:22.901998 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:22.901999 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhwm5\" (UniqueName: \"kubernetes.io/projected/540058b9-e9db-4405-9dd1-03a4f85e50d6-kube-api-access-lhwm5\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:23.064956 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:23.064926 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl"] Oct 30 19:08:23.067009 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:08:23.066978 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod194c0736_90dd_41b0_b316_9858113b2477.slice/crio-ec70e644050ada76b1ef9c73b4379f43df456ed1e769be08bb24fc3a29a8a0f1 WatchSource:0}: Error finding container ec70e644050ada76b1ef9c73b4379f43df456ed1e769be08bb24fc3a29a8a0f1: Status 404 returned error can't find the container with id ec70e644050ada76b1ef9c73b4379f43df456ed1e769be08bb24fc3a29a8a0f1 Oct 30 19:08:23.081208 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:23.081177 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs"] Oct 30 19:08:23.081666 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:23.081652 4531 cpu_manager.go:414] "containerMap: removing container" podUID="540058b9-e9db-4405-9dd1-03a4f85e50d6" containerName="revoke-keys" Oct 30 19:08:23.081708 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:23.081668 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="540058b9-e9db-4405-9dd1-03a4f85e50d6" containerName="revoke-keys" Oct 30 19:08:23.081793 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:23.081783 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="540058b9-e9db-4405-9dd1-03a4f85e50d6" containerName="revoke-keys" Oct 30 19:08:23.086321 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:23.086298 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs" Oct 30 19:08:23.148912 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:23.148880 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs"] Oct 30 19:08:23.204760 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:23.204717 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx562\" (UniqueName: \"kubernetes.io/projected/a9ace4a6-85c4-477a-99d5-0995681b0bb7-kube-api-access-sx562\") pod \"payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs\" (UID: \"a9ace4a6-85c4-477a-99d5-0995681b0bb7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs" Oct 30 19:08:23.204951 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:23.204857 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ace4a6-85c4-477a-99d5-0995681b0bb7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs\" (UID: \"a9ace4a6-85c4-477a-99d5-0995681b0bb7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs" Oct 30 19:08:23.306716 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:23.306671 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ace4a6-85c4-477a-99d5-0995681b0bb7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs\" (UID: \"a9ace4a6-85c4-477a-99d5-0995681b0bb7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs" Oct 30 19:08:23.306895 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:23.306814 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx562\" (UniqueName: \"kubernetes.io/projected/a9ace4a6-85c4-477a-99d5-0995681b0bb7-kube-api-access-sx562\") pod \"payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs\" (UID: \"a9ace4a6-85c4-477a-99d5-0995681b0bb7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs" Oct 30 19:08:23.307460 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:23.307440 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ace4a6-85c4-477a-99d5-0995681b0bb7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs\" (UID: \"a9ace4a6-85c4-477a-99d5-0995681b0bb7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs" Oct 30 19:08:23.368829 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:23.368744 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx562\" (UniqueName: \"kubernetes.io/projected/a9ace4a6-85c4-477a-99d5-0995681b0bb7-kube-api-access-sx562\") pod \"payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs\" (UID: \"a9ace4a6-85c4-477a-99d5-0995681b0bb7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs" Oct 30 19:08:23.411698 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:23.411650 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs" Oct 30 19:08:23.547397 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:23.547362 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl" event={"ID":"194c0736-90dd-41b0-b316-9858113b2477","Type":"ContainerStarted","Data":"ce5d5826f2eff92976a4c76fc110ee9b7407386c0a4f043ffa2dac0551b42b31"} Oct 30 19:08:23.547918 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:23.547427 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl" event={"ID":"194c0736-90dd-41b0-b316-9858113b2477","Type":"ContainerStarted","Data":"ec70e644050ada76b1ef9c73b4379f43df456ed1e769be08bb24fc3a29a8a0f1"} Oct 30 19:08:23.547918 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:23.547547 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl" Oct 30 19:08:23.549081 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:23.549059 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz" Oct 30 19:08:23.549250 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:23.549105 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz" event={"ID":"540058b9-e9db-4405-9dd1-03a4f85e50d6","Type":"ContainerDied","Data":"e9bf2ce68fc78cd7dbfacdf2c4a0bae13f21c53bad66df27eb80e6a802c68dbb"} Oct 30 19:08:23.549250 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:23.549192 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9bf2ce68fc78cd7dbfacdf2c4a0bae13f21c53bad66df27eb80e6a802c68dbb" Oct 30 19:08:23.667731 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:23.667605 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl" podStartSLOduration=1.667584167 podStartE2EDuration="1.667584167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:23.666313055 +0000 UTC m=+2063.693418014" watchObservedRunningTime="2026-10-30 19:08:23.667584167 +0000 UTC m=+2063.694689129" Oct 30 19:08:23.803453 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:23.803412 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs"] Oct 30 19:08:23.816259 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:08:23.816225 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ace4a6_85c4_477a_99d5_0995681b0bb7.slice/crio-c0373d6f71ad71d9417d6bc7540e928134e29ee48df658c1df80529af8fc8f27 WatchSource:0}: Error finding container c0373d6f71ad71d9417d6bc7540e928134e29ee48df658c1df80529af8fc8f27: Status 404 returned error can't find the container with id c0373d6f71ad71d9417d6bc7540e928134e29ee48df658c1df80529af8fc8f27 Oct 30 19:08:24.557537 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:24.557502 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs" event={"ID":"a9ace4a6-85c4-477a-99d5-0995681b0bb7","Type":"ContainerStarted","Data":"d9d653d0c5d65caafa98474b437654c31be00d48a751629bcd06040c0aa6939d"} Oct 30 19:08:24.558001 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:24.557820 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs" Oct 30 19:08:24.558001 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:24.557857 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs" event={"ID":"a9ace4a6-85c4-477a-99d5-0995681b0bb7","Type":"ContainerStarted","Data":"c0373d6f71ad71d9417d6bc7540e928134e29ee48df658c1df80529af8fc8f27"} Oct 30 19:08:24.677435 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:24.677373 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs" podStartSLOduration=1.677346042 podStartE2EDuration="1.677346042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:24.67381636 +0000 UTC m=+2064.700921320" watchObservedRunningTime="2026-10-30 19:08:24.677346042 +0000 UTC m=+2064.704451001" Oct 30 19:08:26.398442 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:08:26.398397 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:08:34.560601 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:34.560564 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl" Oct 30 19:08:34.923204 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:34.923091 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2"] Oct 30 19:08:34.927730 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:34.927698 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2" Oct 30 19:08:34.958393 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:34.958344 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw95r\" (UniqueName: \"kubernetes.io/projected/d52c191a-2663-4f13-8384-368d5b97c780-kube-api-access-qw95r\") pod \"payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2\" (UID: \"d52c191a-2663-4f13-8384-368d5b97c780\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2" Oct 30 19:08:34.958572 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:34.958532 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52c191a-2663-4f13-8384-368d5b97c780-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2\" (UID: \"d52c191a-2663-4f13-8384-368d5b97c780\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2" Oct 30 19:08:34.982844 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:34.982810 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0ee379\"" Oct 30 19:08:34.983211 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:34.983049 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0ee379-dockercfg-4vpdm\"" Oct 30 19:08:34.985687 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:34.985655 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2"] Oct 30 19:08:35.059531 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:35.059476 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw95r\" (UniqueName: \"kubernetes.io/projected/d52c191a-2663-4f13-8384-368d5b97c780-kube-api-access-qw95r\") pod \"payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2\" (UID: \"d52c191a-2663-4f13-8384-368d5b97c780\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2" Oct 30 19:08:35.059747 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:35.059628 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52c191a-2663-4f13-8384-368d5b97c780-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2\" (UID: \"d52c191a-2663-4f13-8384-368d5b97c780\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2" Oct 30 19:08:35.060458 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:35.060429 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52c191a-2663-4f13-8384-368d5b97c780-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2\" (UID: \"d52c191a-2663-4f13-8384-368d5b97c780\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2" Oct 30 19:08:35.121066 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:35.121021 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw95r\" (UniqueName: \"kubernetes.io/projected/d52c191a-2663-4f13-8384-368d5b97c780-kube-api-access-qw95r\") pod \"payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2\" (UID: \"d52c191a-2663-4f13-8384-368d5b97c780\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2" Oct 30 19:08:35.242378 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:35.242340 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2" Oct 30 19:08:35.566769 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:35.566689 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs" Oct 30 19:08:35.624902 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:35.624759 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2"] Oct 30 19:08:35.628225 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:08:35.628192 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52c191a_2663_4f13_8384_368d5b97c780.slice/crio-c351f9f1167a98a465d1895a529c949ed1d2395cba2a0f97b25018b193408339 WatchSource:0}: Error finding container c351f9f1167a98a465d1895a529c949ed1d2395cba2a0f97b25018b193408339: Status 404 returned error can't find the container with id c351f9f1167a98a465d1895a529c949ed1d2395cba2a0f97b25018b193408339 Oct 30 19:08:36.631931 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:36.631894 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2" event={"ID":"d52c191a-2663-4f13-8384-368d5b97c780","Type":"ContainerStarted","Data":"32b0e72abd4547cbf069367b3e0ad29670d60e989597e3d1660f0f06d7eee550"} Oct 30 19:08:36.632493 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:36.631959 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2" event={"ID":"d52c191a-2663-4f13-8384-368d5b97c780","Type":"ContainerStarted","Data":"c351f9f1167a98a465d1895a529c949ed1d2395cba2a0f97b25018b193408339"} Oct 30 19:08:36.632493 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:36.631996 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2" Oct 30 19:08:36.751590 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:36.751525 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2" podStartSLOduration=2.751504038 podStartE2EDuration="2.751504038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:36.748425635 +0000 UTC m=+2076.775530595" watchObservedRunningTime="2026-10-30 19:08:36.751504038 +0000 UTC m=+2076.778609000" Oct 30 19:08:45.045847 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:45.045803 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs"] Oct 30 19:08:45.046358 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:45.046080 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" podUID="6f0c5609-7aa9-4360-9818-6bdbc744554a" containerName="istio-proxy" containerID="cri-o://e644fefc8000830a5149b6ac0d5718549d3bb1a3e2ded1428c20ebda98406dde" gracePeriod=30 Oct 30 19:08:45.830582 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:45.830546 4531 patch_prober.go:29] interesting pod/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:08:45.830752 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:45.830615 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" podUID="6f0c5609-7aa9-4360-9818-6bdbc744554a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:08:47.640304 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:47.640267 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2" Oct 30 19:08:47.870808 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:47.870775 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9"] Oct 30 19:08:47.871038 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:47.871010 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" podUID="7d74e373-44cb-4895-b98f-7efa9be01b7d" containerName="istio-proxy" containerID="cri-o://fb4efa242bc08b5c770431eabef642fe8223899960947925df2ebc7180183b6e" gracePeriod=30 Oct 30 19:08:49.014381 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.014343 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx"] Oct 30 19:08:49.019663 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.019636 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx" Oct 30 19:08:49.063862 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.063825 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s"] Oct 30 19:08:49.064377 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.064327 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" podUID="ce1788f5-f0c0-4457-a310-694f47a97d2c" containerName="istio-proxy" containerID="cri-o://16f8c91eb59437589ba4a871644f124d24837dbc9bd29e2735365d9b05e9f6e0" gracePeriod=30 Oct 30 19:08:49.092913 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.092875 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7f44c7\"" Oct 30 19:08:49.093087 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.092944 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7f44c7-dockercfg-qrv7x\"" Oct 30 19:08:49.095775 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.095740 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx"] Oct 30 19:08:49.158094 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.158061 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v6cc\" (UniqueName: \"kubernetes.io/projected/cefd3d24-b565-4382-86cd-f408ccacf5d9-kube-api-access-4v6cc\") pod \"payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx\" (UID: \"cefd3d24-b565-4382-86cd-f408ccacf5d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx" Oct 30 19:08:49.158286 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.158113 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cefd3d24-b565-4382-86cd-f408ccacf5d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx\" (UID: \"cefd3d24-b565-4382-86cd-f408ccacf5d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx" Oct 30 19:08:49.162121 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.162097 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt"] Oct 30 19:08:49.165938 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.165922 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt" Oct 30 19:08:49.229803 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.229766 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt"] Oct 30 19:08:49.259660 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.259619 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cefd3d24-b565-4382-86cd-f408ccacf5d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx\" (UID: \"cefd3d24-b565-4382-86cd-f408ccacf5d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx" Oct 30 19:08:49.259866 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.259706 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f5e516b-073c-4d78-a0e8-5fbf82b18b93-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt\" (UID: \"7f5e516b-073c-4d78-a0e8-5fbf82b18b93\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt" Oct 30 19:08:49.259866 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.259805 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9slt\" (UniqueName: \"kubernetes.io/projected/7f5e516b-073c-4d78-a0e8-5fbf82b18b93-kube-api-access-l9slt\") pod \"payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt\" (UID: \"7f5e516b-073c-4d78-a0e8-5fbf82b18b93\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt" Oct 30 19:08:49.260013 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.259946 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v6cc\" (UniqueName: \"kubernetes.io/projected/cefd3d24-b565-4382-86cd-f408ccacf5d9-kube-api-access-4v6cc\") pod \"payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx\" (UID: \"cefd3d24-b565-4382-86cd-f408ccacf5d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx" Oct 30 19:08:49.261464 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.260514 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cefd3d24-b565-4382-86cd-f408ccacf5d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx\" (UID: \"cefd3d24-b565-4382-86cd-f408ccacf5d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx" Oct 30 19:08:49.322304 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.322237 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v6cc\" (UniqueName: \"kubernetes.io/projected/cefd3d24-b565-4382-86cd-f408ccacf5d9-kube-api-access-4v6cc\") pod \"payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx\" (UID: \"cefd3d24-b565-4382-86cd-f408ccacf5d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx" Oct 30 19:08:49.340401 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.340359 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx" Oct 30 19:08:49.360842 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.360803 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9slt\" (UniqueName: \"kubernetes.io/projected/7f5e516b-073c-4d78-a0e8-5fbf82b18b93-kube-api-access-l9slt\") pod \"payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt\" (UID: \"7f5e516b-073c-4d78-a0e8-5fbf82b18b93\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt" Oct 30 19:08:49.361109 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.360957 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f5e516b-073c-4d78-a0e8-5fbf82b18b93-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt\" (UID: \"7f5e516b-073c-4d78-a0e8-5fbf82b18b93\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt" Oct 30 19:08:49.362099 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.362028 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f5e516b-073c-4d78-a0e8-5fbf82b18b93-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt\" (UID: \"7f5e516b-073c-4d78-a0e8-5fbf82b18b93\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt" Oct 30 19:08:49.423048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.423013 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9slt\" (UniqueName: \"kubernetes.io/projected/7f5e516b-073c-4d78-a0e8-5fbf82b18b93-kube-api-access-l9slt\") pod \"payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt\" (UID: \"7f5e516b-073c-4d78-a0e8-5fbf82b18b93\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt" Oct 30 19:08:49.480740 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.480700 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt" Oct 30 19:08:49.572538 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.572469 4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-66db998887-wqgt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:08:49.572538 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.572520 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" podUID="7d74e373-44cb-4895-b98f-7efa9be01b7d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:08:49.739656 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.739626 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx"] Oct 30 19:08:49.741998 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:08:49.741961 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcefd3d24_b565_4382_86cd_f408ccacf5d9.slice/crio-3d4b125bc36e693cfdaf56575dcd74cfa1e13c4d938cfc7816db71e6a9563bea WatchSource:0}: Error finding container 3d4b125bc36e693cfdaf56575dcd74cfa1e13c4d938cfc7816db71e6a9563bea: Status 404 returned error can't find the container with id 3d4b125bc36e693cfdaf56575dcd74cfa1e13c4d938cfc7816db71e6a9563bea Oct 30 19:08:49.752610 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.752586 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx" event={"ID":"cefd3d24-b565-4382-86cd-f408ccacf5d9","Type":"ContainerStarted","Data":"3d4b125bc36e693cfdaf56575dcd74cfa1e13c4d938cfc7816db71e6a9563bea"} Oct 30 19:08:49.876597 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:49.876489 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt"] Oct 30 19:08:49.879398 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:08:49.879366 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5e516b_073c_4d78_a0e8_5fbf82b18b93.slice/crio-c7fbfd0da0dcd94be1d378e96eaa664715cc308ee66a22ade468bc39153370e3 WatchSource:0}: Error finding container c7fbfd0da0dcd94be1d378e96eaa664715cc308ee66a22ade468bc39153370e3: Status 404 returned error can't find the container with id c7fbfd0da0dcd94be1d378e96eaa664715cc308ee66a22ade468bc39153370e3 Oct 30 19:08:50.299089 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.299064 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 19:08:50.475085 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.474984 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-token\" (UniqueName: \"kubernetes.io/projected/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-token\") pod \"6f0c5609-7aa9-4360-9818-6bdbc744554a\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " Oct 30 19:08:50.475085 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.475079 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-credential-socket\") pod \"6f0c5609-7aa9-4360-9818-6bdbc744554a\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " Oct 30 19:08:50.475381 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.475156 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-envoy\") pod \"6f0c5609-7aa9-4360-9818-6bdbc744554a\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " Oct 30 19:08:50.475381 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.475219 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-data\") pod \"6f0c5609-7aa9-4360-9818-6bdbc744554a\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " Oct 30 19:08:50.475381 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.475301 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-podinfo\") pod \"6f0c5609-7aa9-4360-9818-6bdbc744554a\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " Oct 30 19:08:50.475381 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.475365 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-workload-socket\") pod \"6f0c5609-7aa9-4360-9818-6bdbc744554a\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " Oct 30 19:08:50.475573 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.475404 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f0c5609-7aa9-4360-9818-6bdbc744554a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0c5609-7aa9-4360-9818-6bdbc744554a-istiod-ca-cert\") pod \"6f0c5609-7aa9-4360-9818-6bdbc744554a\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " Oct 30 19:08:50.475573 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.475459 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-credential-socket" pod "6f0c5609-7aa9-4360-9818-6bdbc744554a" (UID: "6f0c5609-7aa9-4360-9818-6bdbc744554a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:50.475573 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.475561 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-workload-certs\") pod \"6f0c5609-7aa9-4360-9818-6bdbc744554a\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " Oct 30 19:08:50.475734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.475621 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0c5609-7aa9-4360-9818-6bdbc744554a-kube-api-access-79t7k\" (UniqueName: \"kubernetes.io/projected/6f0c5609-7aa9-4360-9818-6bdbc744554a-kube-api-access-79t7k\") pod \"6f0c5609-7aa9-4360-9818-6bdbc744554a\" (UID: \"6f0c5609-7aa9-4360-9818-6bdbc744554a\") " Oct 30 19:08:50.475870 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.475848 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-data" pod "6f0c5609-7aa9-4360-9818-6bdbc744554a" (UID: "6f0c5609-7aa9-4360-9818-6bdbc744554a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:50.475974 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.475944 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-workload-certs" pod "6f0c5609-7aa9-4360-9818-6bdbc744554a" (UID: "6f0c5609-7aa9-4360-9818-6bdbc744554a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:50.476120 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.476100 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f0c5609-7aa9-4360-9818-6bdbc744554a-istiod-ca-cert" pod "6f0c5609-7aa9-4360-9818-6bdbc744554a" (UID: "6f0c5609-7aa9-4360-9818-6bdbc744554a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:50.476274 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.476254 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:50.476370 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.476281 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:50.476458 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.476382 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-workload-socket" pod "6f0c5609-7aa9-4360-9818-6bdbc744554a" (UID: "6f0c5609-7aa9-4360-9818-6bdbc744554a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:50.476458 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.476406 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:50.476458 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.476424 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f0c5609-7aa9-4360-9818-6bdbc744554a-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:50.477873 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.477850 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-token" pod "6f0c5609-7aa9-4360-9818-6bdbc744554a" (UID: "6f0c5609-7aa9-4360-9818-6bdbc744554a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:50.478646 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.478621 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0c5609-7aa9-4360-9818-6bdbc744554a-kube-api-access-79t7k" pod "6f0c5609-7aa9-4360-9818-6bdbc744554a" (UID: "6f0c5609-7aa9-4360-9818-6bdbc744554a"). InnerVolumeSpecName "kube-api-access-79t7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:50.478767 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.478707 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-podinfo" pod "6f0c5609-7aa9-4360-9818-6bdbc744554a" (UID: "6f0c5609-7aa9-4360-9818-6bdbc744554a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:08:50.479793 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.479762 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-envoy" pod "6f0c5609-7aa9-4360-9818-6bdbc744554a" (UID: "6f0c5609-7aa9-4360-9818-6bdbc744554a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:50.577555 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.577514 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:50.577555 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.577547 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:50.577555 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.577557 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f0c5609-7aa9-4360-9818-6bdbc744554a-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:50.577555 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.577566 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f0c5609-7aa9-4360-9818-6bdbc744554a-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:50.577833 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.577579 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79t7k\" (UniqueName: \"kubernetes.io/projected/6f0c5609-7aa9-4360-9818-6bdbc744554a-kube-api-access-79t7k\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:50.760511 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.760477 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt" event={"ID":"7f5e516b-073c-4d78-a0e8-5fbf82b18b93","Type":"ContainerStarted","Data":"b066d6d5b5dbfc9b3e6bbec80d2b466c6345d9983768a9b8b55387bc98d8f240"} Oct 30 19:08:50.760707 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.760528 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt" event={"ID":"7f5e516b-073c-4d78-a0e8-5fbf82b18b93","Type":"ContainerStarted","Data":"c7fbfd0da0dcd94be1d378e96eaa664715cc308ee66a22ade468bc39153370e3"} Oct 30 19:08:50.760707 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.760665 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt" Oct 30 19:08:50.762284 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.762249 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx" event={"ID":"cefd3d24-b565-4382-86cd-f408ccacf5d9","Type":"ContainerStarted","Data":"71370eeec09f0a0e51054d1d672be19ae08415e32ce04646bdc29a21a082c239"} Oct 30 19:08:50.762423 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.762342 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx" Oct 30 19:08:50.763559 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.763538 4531 generic.go:356] "Generic (PLEG): container finished" podID="6f0c5609-7aa9-4360-9818-6bdbc744554a" containerID="e644fefc8000830a5149b6ac0d5718549d3bb1a3e2ded1428c20ebda98406dde" exitCode=0 Oct 30 19:08:50.763669 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.763579 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" Oct 30 19:08:50.763669 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.763595 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" event={"ID":"6f0c5609-7aa9-4360-9818-6bdbc744554a","Type":"ContainerDied","Data":"e644fefc8000830a5149b6ac0d5718549d3bb1a3e2ded1428c20ebda98406dde"} Oct 30 19:08:50.763669 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.763646 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs" event={"ID":"6f0c5609-7aa9-4360-9818-6bdbc744554a","Type":"ContainerDied","Data":"c58935d044404fc069ac3e82ec867364068e07d064afdafdb0472c9c274f2aad"} Oct 30 19:08:50.763796 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.763677 4531 scope.go:122] "RemoveContainer" containerID="e644fefc8000830a5149b6ac0d5718549d3bb1a3e2ded1428c20ebda98406dde" Oct 30 19:08:50.777953 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.777928 4531 scope.go:122] "RemoveContainer" containerID="e644fefc8000830a5149b6ac0d5718549d3bb1a3e2ded1428c20ebda98406dde" Oct 30 19:08:50.778359 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:08:50.778336 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e644fefc8000830a5149b6ac0d5718549d3bb1a3e2ded1428c20ebda98406dde\": container with ID starting with e644fefc8000830a5149b6ac0d5718549d3bb1a3e2ded1428c20ebda98406dde not found: ID does not exist" containerID="e644fefc8000830a5149b6ac0d5718549d3bb1a3e2ded1428c20ebda98406dde" Oct 30 19:08:50.778440 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.778370 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e644fefc8000830a5149b6ac0d5718549d3bb1a3e2ded1428c20ebda98406dde"} err="failed to get container status \"e644fefc8000830a5149b6ac0d5718549d3bb1a3e2ded1428c20ebda98406dde\": rpc error: code = NotFound desc = could not find container \"e644fefc8000830a5149b6ac0d5718549d3bb1a3e2ded1428c20ebda98406dde\": container with ID starting with e644fefc8000830a5149b6ac0d5718549d3bb1a3e2ded1428c20ebda98406dde not found: ID does not exist" Oct 30 19:08:50.879040 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.878973 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt" podStartSLOduration=1.87895689 podStartE2EDuration="1.87895689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:50.876270805 +0000 UTC m=+2090.903375765" watchObservedRunningTime="2026-10-30 19:08:50.87895689 +0000 UTC m=+2090.906061894" Oct 30 19:08:50.994551 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:50.993071 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx" podStartSLOduration=2.993051251 podStartE2EDuration="2.993051251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:50.992546959 +0000 UTC m=+2091.019651921" watchObservedRunningTime="2026-10-30 19:08:50.993051251 +0000 UTC m=+2091.020156213" Oct 30 19:08:51.106682 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:51.106588 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs"] Oct 30 19:08:51.111306 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:51.111274 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3b6013-openshift-default-85cf856477-8zdcs"] Oct 30 19:08:52.098969 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:52.098935 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f0c5609-7aa9-4360-9818-6bdbc744554a" path="/var/lib/kubelet/pods/6f0c5609-7aa9-4360-9818-6bdbc744554a/volumes" Oct 30 19:08:52.595984 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:52.595944 4531 patch_prober.go:29] interesting pod/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:08:52.596158 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:52.596006 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" podUID="ce1788f5-f0c0-4457-a310-694f47a97d2c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:08:53.160425 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.160396 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 19:08:53.310960 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.310924 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-workload-socket\") pod \"7d74e373-44cb-4895-b98f-7efa9be01b7d\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " Oct 30 19:08:53.310960 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.310973 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-token\" (UniqueName: \"kubernetes.io/projected/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-token\") pod \"7d74e373-44cb-4895-b98f-7efa9be01b7d\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " Oct 30 19:08:53.311266 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.311011 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-podinfo\") pod \"7d74e373-44cb-4895-b98f-7efa9be01b7d\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " Oct 30 19:08:53.311266 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.311033 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d74e373-44cb-4895-b98f-7efa9be01b7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d74e373-44cb-4895-b98f-7efa9be01b7d-istiod-ca-cert\") pod \"7d74e373-44cb-4895-b98f-7efa9be01b7d\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " Oct 30 19:08:53.311266 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.311083 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-credential-socket\") pod \"7d74e373-44cb-4895-b98f-7efa9be01b7d\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " Oct 30 19:08:53.311266 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.311115 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d74e373-44cb-4895-b98f-7efa9be01b7d-kube-api-access-xrjsk\" (UniqueName: \"kubernetes.io/projected/7d74e373-44cb-4895-b98f-7efa9be01b7d-kube-api-access-xrjsk\") pod \"7d74e373-44cb-4895-b98f-7efa9be01b7d\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " Oct 30 19:08:53.311266 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.311219 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-workload-socket" pod "7d74e373-44cb-4895-b98f-7efa9be01b7d" (UID: "7d74e373-44cb-4895-b98f-7efa9be01b7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:53.311266 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.311248 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-workload-certs\") pod \"7d74e373-44cb-4895-b98f-7efa9be01b7d\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " Oct 30 19:08:53.311626 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.311306 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-data\") pod \"7d74e373-44cb-4895-b98f-7efa9be01b7d\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " Oct 30 19:08:53.311626 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.311408 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-credential-socket" pod "7d74e373-44cb-4895-b98f-7efa9be01b7d" (UID: "7d74e373-44cb-4895-b98f-7efa9be01b7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:53.311626 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.311440 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-envoy\") pod \"7d74e373-44cb-4895-b98f-7efa9be01b7d\" (UID: \"7d74e373-44cb-4895-b98f-7efa9be01b7d\") " Oct 30 19:08:53.311626 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.311584 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-workload-certs" pod "7d74e373-44cb-4895-b98f-7efa9be01b7d" (UID: "7d74e373-44cb-4895-b98f-7efa9be01b7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:53.311626 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.311611 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-data" pod "7d74e373-44cb-4895-b98f-7efa9be01b7d" (UID: "7d74e373-44cb-4895-b98f-7efa9be01b7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:53.311981 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.311954 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:53.312056 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.311990 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:53.312056 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.312007 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:53.312056 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.312022 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:53.312248 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.312221 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d74e373-44cb-4895-b98f-7efa9be01b7d-istiod-ca-cert" pod "7d74e373-44cb-4895-b98f-7efa9be01b7d" (UID: "7d74e373-44cb-4895-b98f-7efa9be01b7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:53.313985 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.313960 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-podinfo" pod "7d74e373-44cb-4895-b98f-7efa9be01b7d" (UID: "7d74e373-44cb-4895-b98f-7efa9be01b7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:08:53.314236 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.314104 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d74e373-44cb-4895-b98f-7efa9be01b7d-kube-api-access-xrjsk" pod "7d74e373-44cb-4895-b98f-7efa9be01b7d" (UID: "7d74e373-44cb-4895-b98f-7efa9be01b7d"). InnerVolumeSpecName "kube-api-access-xrjsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:53.314236 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.314177 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-token" pod "7d74e373-44cb-4895-b98f-7efa9be01b7d" (UID: "7d74e373-44cb-4895-b98f-7efa9be01b7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:53.314379 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.314340 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-envoy" pod "7d74e373-44cb-4895-b98f-7efa9be01b7d" (UID: "7d74e373-44cb-4895-b98f-7efa9be01b7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:53.412844 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.412754 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:53.412844 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.412789 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d74e373-44cb-4895-b98f-7efa9be01b7d-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:53.412844 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.412799 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrjsk\" (UniqueName: \"kubernetes.io/projected/7d74e373-44cb-4895-b98f-7efa9be01b7d-kube-api-access-xrjsk\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:53.412844 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.412814 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:53.412844 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.412828 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d74e373-44cb-4895-b98f-7efa9be01b7d-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:53.785187 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.785152 4531 generic.go:356] "Generic (PLEG): container finished" podID="7d74e373-44cb-4895-b98f-7efa9be01b7d" containerID="fb4efa242bc08b5c770431eabef642fe8223899960947925df2ebc7180183b6e" exitCode=0 Oct 30 19:08:53.785403 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.785225 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" Oct 30 19:08:53.785403 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.785250 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" event={"ID":"7d74e373-44cb-4895-b98f-7efa9be01b7d","Type":"ContainerDied","Data":"fb4efa242bc08b5c770431eabef642fe8223899960947925df2ebc7180183b6e"} Oct 30 19:08:53.785403 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.785304 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9" event={"ID":"7d74e373-44cb-4895-b98f-7efa9be01b7d","Type":"ContainerDied","Data":"f5bcbe5aac3111182e359fe224f9775381e7a91bccd33a57dce139d120083fbd"} Oct 30 19:08:53.785403 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.785325 4531 scope.go:122] "RemoveContainer" containerID="fb4efa242bc08b5c770431eabef642fe8223899960947925df2ebc7180183b6e" Oct 30 19:08:53.797247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.797222 4531 scope.go:122] "RemoveContainer" containerID="fb4efa242bc08b5c770431eabef642fe8223899960947925df2ebc7180183b6e" Oct 30 19:08:53.797636 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:08:53.797614 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4efa242bc08b5c770431eabef642fe8223899960947925df2ebc7180183b6e\": container with ID starting with fb4efa242bc08b5c770431eabef642fe8223899960947925df2ebc7180183b6e not found: ID does not exist" containerID="fb4efa242bc08b5c770431eabef642fe8223899960947925df2ebc7180183b6e" Oct 30 19:08:53.797733 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.797644 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4efa242bc08b5c770431eabef642fe8223899960947925df2ebc7180183b6e"} err="failed to get container status \"fb4efa242bc08b5c770431eabef642fe8223899960947925df2ebc7180183b6e\": rpc error: code = NotFound desc = could not find container \"fb4efa242bc08b5c770431eabef642fe8223899960947925df2ebc7180183b6e\": container with ID starting with fb4efa242bc08b5c770431eabef642fe8223899960947925df2ebc7180183b6e not found: ID does not exist" Oct 30 19:08:53.950261 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.950216 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9"] Oct 30 19:08:53.999542 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:53.999505 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-wqgt9"] Oct 30 19:08:54.099008 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.098979 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d74e373-44cb-4895-b98f-7efa9be01b7d" path="/var/lib/kubelet/pods/7d74e373-44cb-4895-b98f-7efa9be01b7d/volumes" Oct 30 19:08:54.343679 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.343649 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 19:08:54.424389 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.424353 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce1788f5-f0c0-4457-a310-694f47a97d2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1788f5-f0c0-4457-a310-694f47a97d2c-istiod-ca-cert\") pod \"ce1788f5-f0c0-4457-a310-694f47a97d2c\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " Oct 30 19:08:54.424389 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.424397 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-data\") pod \"ce1788f5-f0c0-4457-a310-694f47a97d2c\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " Oct 30 19:08:54.424630 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.424454 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-podinfo\") pod \"ce1788f5-f0c0-4457-a310-694f47a97d2c\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " Oct 30 19:08:54.424630 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.424478 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-workload-certs\") pod \"ce1788f5-f0c0-4457-a310-694f47a97d2c\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " Oct 30 19:08:54.424630 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.424589 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-workload-socket\") pod \"ce1788f5-f0c0-4457-a310-694f47a97d2c\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " Oct 30 19:08:54.424795 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.424685 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-envoy\") pod \"ce1788f5-f0c0-4457-a310-694f47a97d2c\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " Oct 30 19:08:54.424795 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.424741 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce1788f5-f0c0-4457-a310-694f47a97d2c-kube-api-access-4mnqc\" (UniqueName: \"kubernetes.io/projected/ce1788f5-f0c0-4457-a310-694f47a97d2c-kube-api-access-4mnqc\") pod \"ce1788f5-f0c0-4457-a310-694f47a97d2c\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " Oct 30 19:08:54.424893 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.424799 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-data" pod "ce1788f5-f0c0-4457-a310-694f47a97d2c" (UID: "ce1788f5-f0c0-4457-a310-694f47a97d2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:54.424941 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.424893 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-workload-socket" pod "ce1788f5-f0c0-4457-a310-694f47a97d2c" (UID: "ce1788f5-f0c0-4457-a310-694f47a97d2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:54.424941 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.424923 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce1788f5-f0c0-4457-a310-694f47a97d2c-istiod-ca-cert" pod "ce1788f5-f0c0-4457-a310-694f47a97d2c" (UID: "ce1788f5-f0c0-4457-a310-694f47a97d2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:54.424941 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.424928 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-credential-socket\") pod \"ce1788f5-f0c0-4457-a310-694f47a97d2c\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " Oct 30 19:08:54.425092 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.425004 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-token\" (UniqueName: \"kubernetes.io/projected/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-token\") pod \"ce1788f5-f0c0-4457-a310-694f47a97d2c\" (UID: \"ce1788f5-f0c0-4457-a310-694f47a97d2c\") " Oct 30 19:08:54.425206 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.425158 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-credential-socket" pod "ce1788f5-f0c0-4457-a310-694f47a97d2c" (UID: "ce1788f5-f0c0-4457-a310-694f47a97d2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:54.425206 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.425184 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-workload-certs" pod "ce1788f5-f0c0-4457-a310-694f47a97d2c" (UID: "ce1788f5-f0c0-4457-a310-694f47a97d2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:54.425618 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.425571 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce1788f5-f0c0-4457-a310-694f47a97d2c-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:54.425757 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.425690 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:54.425830 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.425769 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:54.425830 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.425786 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:54.425830 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.425802 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:54.427172 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.427123 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-podinfo" pod "ce1788f5-f0c0-4457-a310-694f47a97d2c" (UID: "ce1788f5-f0c0-4457-a310-694f47a97d2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:08:54.427314 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.427290 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1788f5-f0c0-4457-a310-694f47a97d2c-kube-api-access-4mnqc" pod "ce1788f5-f0c0-4457-a310-694f47a97d2c" (UID: "ce1788f5-f0c0-4457-a310-694f47a97d2c"). InnerVolumeSpecName "kube-api-access-4mnqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:54.427389 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.427343 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-envoy" pod "ce1788f5-f0c0-4457-a310-694f47a97d2c" (UID: "ce1788f5-f0c0-4457-a310-694f47a97d2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:54.427447 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.427395 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-token" pod "ce1788f5-f0c0-4457-a310-694f47a97d2c" (UID: "ce1788f5-f0c0-4457-a310-694f47a97d2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:54.526956 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.526902 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:54.526956 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.526947 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mnqc\" (UniqueName: \"kubernetes.io/projected/ce1788f5-f0c0-4457-a310-694f47a97d2c-kube-api-access-4mnqc\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:54.526956 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.526963 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:54.527258 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.526975 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce1788f5-f0c0-4457-a310-694f47a97d2c-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:08:54.794970 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.794925 4531 generic.go:356] "Generic (PLEG): container finished" podID="ce1788f5-f0c0-4457-a310-694f47a97d2c" containerID="16f8c91eb59437589ba4a871644f124d24837dbc9bd29e2735365d9b05e9f6e0" exitCode=0 Oct 30 19:08:54.795204 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.794989 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" Oct 30 19:08:54.795204 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.795023 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" event={"ID":"ce1788f5-f0c0-4457-a310-694f47a97d2c","Type":"ContainerDied","Data":"16f8c91eb59437589ba4a871644f124d24837dbc9bd29e2735365d9b05e9f6e0"} Oct 30 19:08:54.795204 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.795091 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s" event={"ID":"ce1788f5-f0c0-4457-a310-694f47a97d2c","Type":"ContainerDied","Data":"ae5764b4fe113f7c870869de7eb08a3728fdf305b6d28655d301cb818ba2a9d2"} Oct 30 19:08:54.795204 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.795115 4531 scope.go:122] "RemoveContainer" containerID="16f8c91eb59437589ba4a871644f124d24837dbc9bd29e2735365d9b05e9f6e0" Oct 30 19:08:54.806307 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.806286 4531 scope.go:122] "RemoveContainer" containerID="16f8c91eb59437589ba4a871644f124d24837dbc9bd29e2735365d9b05e9f6e0" Oct 30 19:08:54.806684 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:08:54.806660 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16f8c91eb59437589ba4a871644f124d24837dbc9bd29e2735365d9b05e9f6e0\": container with ID starting with 16f8c91eb59437589ba4a871644f124d24837dbc9bd29e2735365d9b05e9f6e0 not found: ID does not exist" containerID="16f8c91eb59437589ba4a871644f124d24837dbc9bd29e2735365d9b05e9f6e0" Oct 30 19:08:54.806737 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.806696 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16f8c91eb59437589ba4a871644f124d24837dbc9bd29e2735365d9b05e9f6e0"} err="failed to get container status \"16f8c91eb59437589ba4a871644f124d24837dbc9bd29e2735365d9b05e9f6e0\": rpc error: code = NotFound desc = could not find container \"16f8c91eb59437589ba4a871644f124d24837dbc9bd29e2735365d9b05e9f6e0\": container with ID starting with 16f8c91eb59437589ba4a871644f124d24837dbc9bd29e2735365d9b05e9f6e0 not found: ID does not exist" Oct 30 19:08:54.960372 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:54.960339 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s"] Oct 30 19:08:55.010059 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:55.010026 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-8zg5s"] Oct 30 19:08:56.099369 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:08:56.099331 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce1788f5-f0c0-4457-a310-694f47a97d2c" path="/var/lib/kubelet/pods/ce1788f5-f0c0-4457-a310-694f47a97d2c/volumes" Oct 30 19:08:56.398316 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:08:56.398236 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:09:01.626959 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:01.626925 4531 scope.go:122] "RemoveContainer" containerID="13c2c9bc41b55d970049a251471eb060b7c86f2ec3247151b50ef0d5860ea824" Oct 30 19:09:01.774406 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:01.774377 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx" Oct 30 19:09:01.774595 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:01.774454 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt" Oct 30 19:09:16.078578 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:16.078495 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7"] Oct 30 19:09:16.127562 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:16.127526 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-842bbd-zgxw7"] Oct 30 19:09:18.100210 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:18.100174 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2936362b-fefc-4625-bbfc-57da0fb1dae6" path="/var/lib/kubelet/pods/2936362b-fefc-4625-bbfc-57da0fb1dae6/volumes" Oct 30 19:09:24.124526 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.124493 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67448-bc4nw"] Oct 30 19:09:24.125012 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.124997 4531 cpu_manager.go:414] "containerMap: removing container" podUID="6f0c5609-7aa9-4360-9818-6bdbc744554a" containerName="istio-proxy" Oct 30 19:09:24.125059 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.125013 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0c5609-7aa9-4360-9818-6bdbc744554a" containerName="istio-proxy" Oct 30 19:09:24.125059 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.125035 4531 cpu_manager.go:414] "containerMap: removing container" podUID="ce1788f5-f0c0-4457-a310-694f47a97d2c" containerName="istio-proxy" Oct 30 19:09:24.125059 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.125043 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce1788f5-f0c0-4457-a310-694f47a97d2c" containerName="istio-proxy" Oct 30 19:09:24.125201 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.125060 4531 cpu_manager.go:414] "containerMap: removing container" podUID="7d74e373-44cb-4895-b98f-7efa9be01b7d" containerName="istio-proxy" Oct 30 19:09:24.125201 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.125067 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d74e373-44cb-4895-b98f-7efa9be01b7d" containerName="istio-proxy" Oct 30 19:09:24.125201 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.125174 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0c5609-7aa9-4360-9818-6bdbc744554a" containerName="istio-proxy" Oct 30 19:09:24.125201 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.125186 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce1788f5-f0c0-4457-a310-694f47a97d2c" containerName="istio-proxy" Oct 30 19:09:24.125201 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.125193 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d74e373-44cb-4895-b98f-7efa9be01b7d" containerName="istio-proxy" Oct 30 19:09:24.128317 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.128302 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67448-bc4nw" Oct 30 19:09:24.165852 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.165816 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989b7485-a99c-4dae-af11-851e6fd6205b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e67448-bc4nw\" (UID: \"989b7485-a99c-4dae-af11-851e6fd6205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67448-bc4nw" Oct 30 19:09:24.166009 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.165947 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvcjn\" (UniqueName: \"kubernetes.io/projected/989b7485-a99c-4dae-af11-851e6fd6205b-kube-api-access-rvcjn\") pod \"maas-api-revoke-keys-e2e-api-b-e67448-bc4nw\" (UID: \"989b7485-a99c-4dae-af11-851e6fd6205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67448-bc4nw" Oct 30 19:09:24.166111 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.166093 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989b7485-a99c-4dae-af11-851e6fd6205b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e67448-bc4nw\" (UID: \"989b7485-a99c-4dae-af11-851e6fd6205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67448-bc4nw" Oct 30 19:09:24.180710 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.180689 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:09:24.180710 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.180692 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:09:24.180872 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.180861 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:09:24.230410 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.230372 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:09:24.233042 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.233012 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67448-bc4nw"] Oct 30 19:09:24.267487 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.267443 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvcjn\" (UniqueName: \"kubernetes.io/projected/989b7485-a99c-4dae-af11-851e6fd6205b-kube-api-access-rvcjn\") pod \"maas-api-revoke-keys-e2e-api-b-e67448-bc4nw\" (UID: \"989b7485-a99c-4dae-af11-851e6fd6205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67448-bc4nw" Oct 30 19:09:24.267683 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.267542 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989b7485-a99c-4dae-af11-851e6fd6205b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e67448-bc4nw\" (UID: \"989b7485-a99c-4dae-af11-851e6fd6205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67448-bc4nw" Oct 30 19:09:24.267683 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.267583 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989b7485-a99c-4dae-af11-851e6fd6205b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e67448-bc4nw\" (UID: \"989b7485-a99c-4dae-af11-851e6fd6205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67448-bc4nw" Oct 30 19:09:24.268281 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.268256 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989b7485-a99c-4dae-af11-851e6fd6205b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e67448-bc4nw\" (UID: \"989b7485-a99c-4dae-af11-851e6fd6205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67448-bc4nw" Oct 30 19:09:24.270272 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.270244 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989b7485-a99c-4dae-af11-851e6fd6205b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e67448-bc4nw\" (UID: \"989b7485-a99c-4dae-af11-851e6fd6205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67448-bc4nw" Oct 30 19:09:24.328469 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.328438 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvcjn\" (UniqueName: \"kubernetes.io/projected/989b7485-a99c-4dae-af11-851e6fd6205b-kube-api-access-rvcjn\") pod \"maas-api-revoke-keys-e2e-api-b-e67448-bc4nw\" (UID: \"989b7485-a99c-4dae-af11-851e6fd6205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67448-bc4nw" Oct 30 19:09:24.441060 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.440958 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67448-bc4nw" Oct 30 19:09:24.816948 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.816911 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67448-bc4nw"] Oct 30 19:09:24.819100 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:09:24.819069 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989b7485_a99c_4dae_af11_851e6fd6205b.slice/crio-d067d99e11d6ae567d4fd4f69b8c9d8adc993ed1a64f0f79210d2d172bf50166 WatchSource:0}: Error finding container d067d99e11d6ae567d4fd4f69b8c9d8adc993ed1a64f0f79210d2d172bf50166: Status 404 returned error can't find the container with id d067d99e11d6ae567d4fd4f69b8c9d8adc993ed1a64f0f79210d2d172bf50166 Oct 30 19:09:24.984012 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:24.983972 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67448-bc4nw" event={"ID":"989b7485-a99c-4dae-af11-851e6fd6205b","Type":"ContainerStarted","Data":"d067d99e11d6ae567d4fd4f69b8c9d8adc993ed1a64f0f79210d2d172bf50166"} Oct 30 19:09:25.992321 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:25.992289 4531 generic.go:356] "Generic (PLEG): container finished" podID="989b7485-a99c-4dae-af11-851e6fd6205b" containerID="785dfcee4fd680d9b44438cf0f14da3639db220daf24f1ceb59936f0072960a9" exitCode=0 Oct 30 19:09:25.992757 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:25.992332 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67448-bc4nw" event={"ID":"989b7485-a99c-4dae-af11-851e6fd6205b","Type":"ContainerDied","Data":"785dfcee4fd680d9b44438cf0f14da3639db220daf24f1ceb59936f0072960a9"} Oct 30 19:09:26.398591 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:09:26.398499 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:09:27.155842 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:27.155818 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67448-bc4nw" Oct 30 19:09:27.197140 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:27.197108 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/989b7485-a99c-4dae-af11-851e6fd6205b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989b7485-a99c-4dae-af11-851e6fd6205b-maas-api-xks-ca\") pod \"989b7485-a99c-4dae-af11-851e6fd6205b\" (UID: \"989b7485-a99c-4dae-af11-851e6fd6205b\") " Oct 30 19:09:27.197301 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:27.197256 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/989b7485-a99c-4dae-af11-851e6fd6205b-kube-api-access-rvcjn\" (UniqueName: \"kubernetes.io/projected/989b7485-a99c-4dae-af11-851e6fd6205b-kube-api-access-rvcjn\") pod \"989b7485-a99c-4dae-af11-851e6fd6205b\" (UID: \"989b7485-a99c-4dae-af11-851e6fd6205b\") " Oct 30 19:09:27.197383 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:27.197347 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/989b7485-a99c-4dae-af11-851e6fd6205b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989b7485-a99c-4dae-af11-851e6fd6205b-maas-api-openshift-ca\") pod \"989b7485-a99c-4dae-af11-851e6fd6205b\" (UID: \"989b7485-a99c-4dae-af11-851e6fd6205b\") " Oct 30 19:09:27.197733 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:27.197702 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/989b7485-a99c-4dae-af11-851e6fd6205b-maas-api-openshift-ca" pod "989b7485-a99c-4dae-af11-851e6fd6205b" (UID: "989b7485-a99c-4dae-af11-851e6fd6205b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:09:27.197927 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:27.197906 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989b7485-a99c-4dae-af11-851e6fd6205b-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:09:27.199230 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:27.199209 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/989b7485-a99c-4dae-af11-851e6fd6205b-maas-api-xks-ca" pod "989b7485-a99c-4dae-af11-851e6fd6205b" (UID: "989b7485-a99c-4dae-af11-851e6fd6205b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:09:27.199416 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:27.199394 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/989b7485-a99c-4dae-af11-851e6fd6205b-kube-api-access-rvcjn" pod "989b7485-a99c-4dae-af11-851e6fd6205b" (UID: "989b7485-a99c-4dae-af11-851e6fd6205b"). InnerVolumeSpecName "kube-api-access-rvcjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:27.298907 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:27.298798 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvcjn\" (UniqueName: \"kubernetes.io/projected/989b7485-a99c-4dae-af11-851e6fd6205b-kube-api-access-rvcjn\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:09:27.298907 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:27.298837 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989b7485-a99c-4dae-af11-851e6fd6205b-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:09:28.006599 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:28.006565 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67448-bc4nw" Oct 30 19:09:28.006798 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:28.006598 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67448-bc4nw" event={"ID":"989b7485-a99c-4dae-af11-851e6fd6205b","Type":"ContainerDied","Data":"d067d99e11d6ae567d4fd4f69b8c9d8adc993ed1a64f0f79210d2d172bf50166"} Oct 30 19:09:28.006798 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:28.006690 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d067d99e11d6ae567d4fd4f69b8c9d8adc993ed1a64f0f79210d2d172bf50166" Oct 30 19:09:37.700184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:37.700147 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2"] Oct 30 19:09:37.700608 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:37.700363 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2" podUID="d52c191a-2663-4f13-8384-368d5b97c780" containerName="payload-pre-processing" containerID="cri-o://32b0e72abd4547cbf069367b3e0ad29670d60e989597e3d1660f0f06d7eee550" gracePeriod=30 Oct 30 19:09:37.971484 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:37.971456 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2" Oct 30 19:09:38.061997 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:38.061957 4531 generic.go:356] "Generic (PLEG): container finished" podID="d52c191a-2663-4f13-8384-368d5b97c780" containerID="32b0e72abd4547cbf069367b3e0ad29670d60e989597e3d1660f0f06d7eee550" exitCode=0 Oct 30 19:09:38.062222 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:38.062009 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2" Oct 30 19:09:38.062222 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:38.062043 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2" event={"ID":"d52c191a-2663-4f13-8384-368d5b97c780","Type":"ContainerDied","Data":"32b0e72abd4547cbf069367b3e0ad29670d60e989597e3d1660f0f06d7eee550"} Oct 30 19:09:38.062222 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:38.062102 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2" event={"ID":"d52c191a-2663-4f13-8384-368d5b97c780","Type":"ContainerDied","Data":"c351f9f1167a98a465d1895a529c949ed1d2395cba2a0f97b25018b193408339"} Oct 30 19:09:38.062222 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:38.062157 4531 scope.go:122] "RemoveContainer" containerID="32b0e72abd4547cbf069367b3e0ad29670d60e989597e3d1660f0f06d7eee550" Oct 30 19:09:38.072965 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:38.072941 4531 scope.go:122] "RemoveContainer" containerID="32b0e72abd4547cbf069367b3e0ad29670d60e989597e3d1660f0f06d7eee550" Oct 30 19:09:38.073354 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:09:38.073326 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32b0e72abd4547cbf069367b3e0ad29670d60e989597e3d1660f0f06d7eee550\": container with ID starting with 32b0e72abd4547cbf069367b3e0ad29670d60e989597e3d1660f0f06d7eee550 not found: ID does not exist" containerID="32b0e72abd4547cbf069367b3e0ad29670d60e989597e3d1660f0f06d7eee550" Oct 30 19:09:38.073463 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:38.073375 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32b0e72abd4547cbf069367b3e0ad29670d60e989597e3d1660f0f06d7eee550"} err="failed to get container status \"32b0e72abd4547cbf069367b3e0ad29670d60e989597e3d1660f0f06d7eee550\": rpc error: code = NotFound desc = could not find container \"32b0e72abd4547cbf069367b3e0ad29670d60e989597e3d1660f0f06d7eee550\": container with ID starting with 32b0e72abd4547cbf069367b3e0ad29670d60e989597e3d1660f0f06d7eee550 not found: ID does not exist" Oct 30 19:09:38.119361 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:38.119316 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d52c191a-2663-4f13-8384-368d5b97c780-kube-api-access-qw95r\" (UniqueName: \"kubernetes.io/projected/d52c191a-2663-4f13-8384-368d5b97c780-kube-api-access-qw95r\") pod \"d52c191a-2663-4f13-8384-368d5b97c780\" (UID: \"d52c191a-2663-4f13-8384-368d5b97c780\") " Oct 30 19:09:38.119585 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:38.119432 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d52c191a-2663-4f13-8384-368d5b97c780-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52c191a-2663-4f13-8384-368d5b97c780-plugins-config-volume\") pod \"d52c191a-2663-4f13-8384-368d5b97c780\" (UID: \"d52c191a-2663-4f13-8384-368d5b97c780\") " Oct 30 19:09:38.119824 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:38.119799 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52c191a-2663-4f13-8384-368d5b97c780-plugins-config-volume" pod "d52c191a-2663-4f13-8384-368d5b97c780" (UID: "d52c191a-2663-4f13-8384-368d5b97c780"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:09:38.119998 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:38.119981 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52c191a-2663-4f13-8384-368d5b97c780-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:09:38.121867 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:38.121827 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52c191a-2663-4f13-8384-368d5b97c780-kube-api-access-qw95r" pod "d52c191a-2663-4f13-8384-368d5b97c780" (UID: "d52c191a-2663-4f13-8384-368d5b97c780"). InnerVolumeSpecName "kube-api-access-qw95r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:38.221113 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:38.221027 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw95r\" (UniqueName: \"kubernetes.io/projected/d52c191a-2663-4f13-8384-368d5b97c780-kube-api-access-qw95r\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:09:38.482738 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:38.482656 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2"] Oct 30 19:09:38.486958 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:38.486932 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0ee379-6b7558959-946k2"] Oct 30 19:09:39.484701 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:39.484667 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz"] Oct 30 19:09:39.533953 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:39.533922 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ee379-lp5bz"] Oct 30 19:09:40.097587 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:40.097554 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="540058b9-e9db-4405-9dd1-03a4f85e50d6" path="/var/lib/kubelet/pods/540058b9-e9db-4405-9dd1-03a4f85e50d6/volumes" Oct 30 19:09:40.097974 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:40.097961 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d52c191a-2663-4f13-8384-368d5b97c780" path="/var/lib/kubelet/pods/d52c191a-2663-4f13-8384-368d5b97c780/volumes" Oct 30 19:09:56.398180 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:09:56.398118 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:09:56.872238 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:56.872204 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c"] Oct 30 19:09:56.872745 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:56.872725 4531 cpu_manager.go:414] "containerMap: removing container" podUID="989b7485-a99c-4dae-af11-851e6fd6205b" containerName="revoke-keys" Oct 30 19:09:56.872850 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:56.872747 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="989b7485-a99c-4dae-af11-851e6fd6205b" containerName="revoke-keys" Oct 30 19:09:56.872850 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:56.872763 4531 cpu_manager.go:414] "containerMap: removing container" podUID="d52c191a-2663-4f13-8384-368d5b97c780" containerName="payload-pre-processing" Oct 30 19:09:56.872850 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:56.872771 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d52c191a-2663-4f13-8384-368d5b97c780" containerName="payload-pre-processing" Oct 30 19:09:56.873015 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:56.872931 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="989b7485-a99c-4dae-af11-851e6fd6205b" containerName="revoke-keys" Oct 30 19:09:56.873015 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:56.872946 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d52c191a-2663-4f13-8384-368d5b97c780" containerName="payload-pre-processing" Oct 30 19:09:56.876332 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:56.876309 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:56.929040 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:56.929014 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-5a035d-openshift-default-dockercfg-p69m7\"" Oct 30 19:09:56.932592 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:56.932565 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpsgp\" (UniqueName: \"kubernetes.io/projected/cac0c9e8-98a5-49b1-93c4-944bfda920a2-kube-api-access-zpsgp\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:56.932790 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:56.932762 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istiod-ca-cert\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:56.932846 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:56.932809 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-data\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:56.932906 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:56.932859 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-podinfo\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:56.932952 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:56.932909 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-envoy\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:56.933002 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:56.932950 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-workload-certs\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:56.933049 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:56.933035 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-workload-socket\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:56.933089 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:56.933063 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-token\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:56.933168 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:56.933103 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-credential-socket\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:56.977483 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:56.977449 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c"] Oct 30 19:09:57.034547 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:57.034511 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-credential-socket\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:57.034745 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:57.034575 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpsgp\" (UniqueName: \"kubernetes.io/projected/cac0c9e8-98a5-49b1-93c4-944bfda920a2-kube-api-access-zpsgp\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:57.034745 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:57.034653 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istiod-ca-cert\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:57.034745 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:57.034672 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-data\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:57.034745 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:57.034691 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-podinfo\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:57.034745 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:57.034711 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-envoy\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:57.034745 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:57.034741 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-workload-certs\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:57.035497 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:57.034797 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-workload-socket\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:57.035497 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:57.034819 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-token\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:57.035497 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:57.034961 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-credential-socket\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:57.035497 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:57.035188 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-workload-certs\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:57.035497 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:57.035292 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-workload-socket\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:57.035497 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:57.035403 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-data\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:57.035497 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:57.035483 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istiod-ca-cert\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:57.037781 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:57.037755 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-envoy\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:57.037970 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:57.037955 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-podinfo\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:57.096939 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:57.096862 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-token\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:57.096939 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:57.096895 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpsgp\" (UniqueName: \"kubernetes.io/projected/cac0c9e8-98a5-49b1-93c4-944bfda920a2-kube-api-access-zpsgp\") pod \"e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:57.190926 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:57.190853 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:57.629274 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:57.629243 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c"] Oct 30 19:09:57.631647 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:09:57.631617 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac0c9e8_98a5_49b1_93c4_944bfda920a2.slice/crio-1c37c7e0d6f9e7cab10528776008804284d3dfa92bb47e33cdb6cc889bcfe7a7 WatchSource:0}: Error finding container 1c37c7e0d6f9e7cab10528776008804284d3dfa92bb47e33cdb6cc889bcfe7a7: Status 404 returned error can't find the container with id 1c37c7e0d6f9e7cab10528776008804284d3dfa92bb47e33cdb6cc889bcfe7a7 Oct 30 19:09:57.634602 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:57.634564 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:09:57.634693 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:57.634637 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:09:57.634693 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:57.634669 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:09:58.177070 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:58.177037 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" event={"ID":"cac0c9e8-98a5-49b1-93c4-944bfda920a2","Type":"ContainerStarted","Data":"132f38aa4f207a572b9c55dbf6123e70927c6484a46538a435913fec0cb0b204"} Oct 30 19:09:58.177356 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:58.177097 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" event={"ID":"cac0c9e8-98a5-49b1-93c4-944bfda920a2","Type":"ContainerStarted","Data":"1c37c7e0d6f9e7cab10528776008804284d3dfa92bb47e33cdb6cc889bcfe7a7"} Oct 30 19:09:58.191009 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:58.190983 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:58.196768 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:58.196745 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:58.301425 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:58.301353 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" podStartSLOduration=2.301334537 podStartE2EDuration="2.301334537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:58.297695359 +0000 UTC m=+2158.324800318" watchObservedRunningTime="2026-10-30 19:09:58.301334537 +0000 UTC m=+2158.328439496" Oct 30 19:09:59.182808 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:59.182775 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:09:59.183700 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:09:59.183683 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:10:00.272446 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:00.270471 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5"] Oct 30 19:10:00.276456 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:00.276351 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5" Oct 30 19:10:00.321772 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:00.321722 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5"] Oct 30 19:10:00.367753 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:00.367726 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-0be4a4\"" Oct 30 19:10:00.367753 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:00.367749 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-0be4a4-dockercfg-2gp5h\"" Oct 30 19:10:00.373709 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:00.373675 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79nlb\" (UniqueName: \"kubernetes.io/projected/97759a69-ea8f-4f5d-b419-a6c314b0c47d-kube-api-access-79nlb\") pod \"payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5\" (UID: \"97759a69-ea8f-4f5d-b419-a6c314b0c47d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5" Oct 30 19:10:00.373709 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:00.373712 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97759a69-ea8f-4f5d-b419-a6c314b0c47d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5\" (UID: \"97759a69-ea8f-4f5d-b419-a6c314b0c47d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5" Oct 30 19:10:00.475028 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:00.474991 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79nlb\" (UniqueName: \"kubernetes.io/projected/97759a69-ea8f-4f5d-b419-a6c314b0c47d-kube-api-access-79nlb\") pod \"payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5\" (UID: \"97759a69-ea8f-4f5d-b419-a6c314b0c47d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5" Oct 30 19:10:00.475263 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:00.475034 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97759a69-ea8f-4f5d-b419-a6c314b0c47d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5\" (UID: \"97759a69-ea8f-4f5d-b419-a6c314b0c47d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5" Oct 30 19:10:00.475720 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:00.475700 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97759a69-ea8f-4f5d-b419-a6c314b0c47d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5\" (UID: \"97759a69-ea8f-4f5d-b419-a6c314b0c47d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5" Oct 30 19:10:00.537426 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:00.537340 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79nlb\" (UniqueName: \"kubernetes.io/projected/97759a69-ea8f-4f5d-b419-a6c314b0c47d-kube-api-access-79nlb\") pod \"payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5\" (UID: \"97759a69-ea8f-4f5d-b419-a6c314b0c47d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5" Oct 30 19:10:00.594816 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:00.594774 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5" Oct 30 19:10:00.977834 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:00.977812 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5"] Oct 30 19:10:00.979912 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:10:00.979876 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97759a69_ea8f_4f5d_b419_a6c314b0c47d.slice/crio-048e904fd5c7fdc26d91b1c3d2ece32c7466ece36839a1b325cbfb4c71d88d4b WatchSource:0}: Error finding container 048e904fd5c7fdc26d91b1c3d2ece32c7466ece36839a1b325cbfb4c71d88d4b: Status 404 returned error can't find the container with id 048e904fd5c7fdc26d91b1c3d2ece32c7466ece36839a1b325cbfb4c71d88d4b Oct 30 19:10:01.197210 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:01.197177 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5" event={"ID":"97759a69-ea8f-4f5d-b419-a6c314b0c47d","Type":"ContainerStarted","Data":"df5dd55b5b0e3e86fc9fc8a682d34bd4e0c9f7edebd3e76db42f2d13119fe5c1"} Oct 30 19:10:01.197210 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:01.197226 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5" event={"ID":"97759a69-ea8f-4f5d-b419-a6c314b0c47d","Type":"ContainerStarted","Data":"048e904fd5c7fdc26d91b1c3d2ece32c7466ece36839a1b325cbfb4c71d88d4b"} Oct 30 19:10:01.197492 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:01.197328 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5" Oct 30 19:10:01.315227 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:01.315171 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5" podStartSLOduration=1.315157022 podStartE2EDuration="1.315157022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:01.314344953 +0000 UTC m=+2161.341449913" watchObservedRunningTime="2026-10-30 19:10:01.315157022 +0000 UTC m=+2161.342261980" Oct 30 19:10:01.676910 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:01.676818 4531 scope.go:122] "RemoveContainer" containerID="677bca08eefe3781db91acc358fb8aca2766ff6a0604c7f33d4c9e409987d2d4" Oct 30 19:10:07.847781 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:07.847733 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms"] Oct 30 19:10:07.854056 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:07.854016 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms" Oct 30 19:10:07.907367 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:07.907330 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-5a035d-dockercfg-28f8s\"" Oct 30 19:10:07.910165 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:07.910090 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-5a035d\"" Oct 30 19:10:07.910732 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:07.910705 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms"] Oct 30 19:10:07.969241 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:07.969194 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7337eb2d-f4b3-44a4-a2bf-3d7557522984-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms\" (UID: \"7337eb2d-f4b3-44a4-a2bf-3d7557522984\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms" Oct 30 19:10:07.969241 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:07.969246 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb86n\" (UniqueName: \"kubernetes.io/projected/7337eb2d-f4b3-44a4-a2bf-3d7557522984-kube-api-access-hb86n\") pod \"payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms\" (UID: \"7337eb2d-f4b3-44a4-a2bf-3d7557522984\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms" Oct 30 19:10:08.070554 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:08.070504 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7337eb2d-f4b3-44a4-a2bf-3d7557522984-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms\" (UID: \"7337eb2d-f4b3-44a4-a2bf-3d7557522984\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms" Oct 30 19:10:08.070554 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:08.070556 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb86n\" (UniqueName: \"kubernetes.io/projected/7337eb2d-f4b3-44a4-a2bf-3d7557522984-kube-api-access-hb86n\") pod \"payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms\" (UID: \"7337eb2d-f4b3-44a4-a2bf-3d7557522984\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms" Oct 30 19:10:08.071341 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:08.071319 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7337eb2d-f4b3-44a4-a2bf-3d7557522984-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms\" (UID: \"7337eb2d-f4b3-44a4-a2bf-3d7557522984\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms" Oct 30 19:10:08.132577 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:08.132487 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb86n\" (UniqueName: \"kubernetes.io/projected/7337eb2d-f4b3-44a4-a2bf-3d7557522984-kube-api-access-hb86n\") pod \"payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms\" (UID: \"7337eb2d-f4b3-44a4-a2bf-3d7557522984\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms" Oct 30 19:10:08.181287 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:08.181249 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms" Oct 30 19:10:08.568708 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:08.568676 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms"] Oct 30 19:10:08.570752 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:10:08.570718 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7337eb2d_f4b3_44a4_a2bf_3d7557522984.slice/crio-e58f1cd8fcf9cbf8da832369824a07ac8d92a6b63218544f69b127a421a8fa6e WatchSource:0}: Error finding container e58f1cd8fcf9cbf8da832369824a07ac8d92a6b63218544f69b127a421a8fa6e: Status 404 returned error can't find the container with id e58f1cd8fcf9cbf8da832369824a07ac8d92a6b63218544f69b127a421a8fa6e Oct 30 19:10:09.247428 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:09.247384 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms" event={"ID":"7337eb2d-f4b3-44a4-a2bf-3d7557522984","Type":"ContainerStarted","Data":"c9c7226ad12dbfda46788e7aa0e58ae0dcea5446eceef6dc2cda2377b9740009"} Oct 30 19:10:09.247876 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:09.247449 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms" event={"ID":"7337eb2d-f4b3-44a4-a2bf-3d7557522984","Type":"ContainerStarted","Data":"e58f1cd8fcf9cbf8da832369824a07ac8d92a6b63218544f69b127a421a8fa6e"} Oct 30 19:10:09.247876 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:09.247542 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms" Oct 30 19:10:09.365758 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:09.365671 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms" podStartSLOduration=2.365651029 podStartE2EDuration="2.365651029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:09.363178395 +0000 UTC m=+2169.390283363" watchObservedRunningTime="2026-10-30 19:10:09.365651029 +0000 UTC m=+2169.392756050" Oct 30 19:10:12.207651 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:12.207622 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5" Oct 30 19:10:20.257560 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:20.257527 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms" Oct 30 19:10:24.064290 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:24.064248 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4"] Oct 30 19:10:24.144529 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:24.144395 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w"] Oct 30 19:10:25.235377 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:25.235336 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782"] Oct 30 19:10:25.239526 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:25.239497 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782" Oct 30 19:10:25.344945 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:25.344906 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:10:25.344945 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:25.344934 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:10:25.344945 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:25.344906 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:10:25.344945 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:25.344909 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:10:25.347679 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:25.347641 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782"] Oct 30 19:10:25.379020 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:25.378976 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed9ead1f-6b8c-4823-8a20-fdda94abf5bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782\" (UID: \"ed9ead1f-6b8c-4823-8a20-fdda94abf5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782" Oct 30 19:10:25.379020 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:25.379018 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlm6q\" (UniqueName: \"kubernetes.io/projected/ed9ead1f-6b8c-4823-8a20-fdda94abf5bc-kube-api-access-dlm6q\") pod \"maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782\" (UID: \"ed9ead1f-6b8c-4823-8a20-fdda94abf5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782" Oct 30 19:10:25.379357 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:25.379223 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed9ead1f-6b8c-4823-8a20-fdda94abf5bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782\" (UID: \"ed9ead1f-6b8c-4823-8a20-fdda94abf5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782" Oct 30 19:10:25.493212 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:25.487811 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed9ead1f-6b8c-4823-8a20-fdda94abf5bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782\" (UID: \"ed9ead1f-6b8c-4823-8a20-fdda94abf5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782" Oct 30 19:10:25.493212 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:25.487992 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed9ead1f-6b8c-4823-8a20-fdda94abf5bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782\" (UID: \"ed9ead1f-6b8c-4823-8a20-fdda94abf5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782" Oct 30 19:10:25.493212 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:25.488025 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlm6q\" (UniqueName: \"kubernetes.io/projected/ed9ead1f-6b8c-4823-8a20-fdda94abf5bc-kube-api-access-dlm6q\") pod \"maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782\" (UID: \"ed9ead1f-6b8c-4823-8a20-fdda94abf5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782" Oct 30 19:10:25.493212 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:25.489449 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed9ead1f-6b8c-4823-8a20-fdda94abf5bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782\" (UID: \"ed9ead1f-6b8c-4823-8a20-fdda94abf5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782" Oct 30 19:10:25.502650 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:25.502609 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed9ead1f-6b8c-4823-8a20-fdda94abf5bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782\" (UID: \"ed9ead1f-6b8c-4823-8a20-fdda94abf5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782" Oct 30 19:10:25.551505 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:25.551473 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlm6q\" (UniqueName: \"kubernetes.io/projected/ed9ead1f-6b8c-4823-8a20-fdda94abf5bc-kube-api-access-dlm6q\") pod \"maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782\" (UID: \"ed9ead1f-6b8c-4823-8a20-fdda94abf5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782" Oct 30 19:10:25.556323 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:25.556296 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782" Oct 30 19:10:26.000208 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:26.000182 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782"] Oct 30 19:10:26.002248 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:10:26.002215 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded9ead1f_6b8c_4823_8a20_fdda94abf5bc.slice/crio-2e9f547e6a3e3faa19919ca9252c26a0eb3c2e69890798a1be946a3eb6ae9711 WatchSource:0}: Error finding container 2e9f547e6a3e3faa19919ca9252c26a0eb3c2e69890798a1be946a3eb6ae9711: Status 404 returned error can't find the container with id 2e9f547e6a3e3faa19919ca9252c26a0eb3c2e69890798a1be946a3eb6ae9711 Oct 30 19:10:26.351619 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:26.351566 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782" event={"ID":"ed9ead1f-6b8c-4823-8a20-fdda94abf5bc","Type":"ContainerStarted","Data":"c39aaa8435065d3c0ac481ddd426e557deec7c2cde9d3880ce348a918787ab87"} Oct 30 19:10:26.352174 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:26.351635 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782" event={"ID":"ed9ead1f-6b8c-4823-8a20-fdda94abf5bc","Type":"ContainerStarted","Data":"2e9f547e6a3e3faa19919ca9252c26a0eb3c2e69890798a1be946a3eb6ae9711"} Oct 30 19:10:26.402335 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:10:26.402298 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:10:26.467367 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:26.467123 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782" podStartSLOduration=1.467107695 podStartE2EDuration="1.467107695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:26.46643603 +0000 UTC m=+2186.493540990" watchObservedRunningTime="2026-10-30 19:10:26.467107695 +0000 UTC m=+2186.494212674" Oct 30 19:10:27.358771 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:27.358732 4531 generic.go:356] "Generic (PLEG): container finished" podID="ed9ead1f-6b8c-4823-8a20-fdda94abf5bc" containerID="c39aaa8435065d3c0ac481ddd426e557deec7c2cde9d3880ce348a918787ab87" exitCode=0 Oct 30 19:10:27.359219 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:27.358817 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782" event={"ID":"ed9ead1f-6b8c-4823-8a20-fdda94abf5bc","Type":"ContainerDied","Data":"c39aaa8435065d3c0ac481ddd426e557deec7c2cde9d3880ce348a918787ab87"} Oct 30 19:10:28.530294 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:28.530262 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782" Oct 30 19:10:28.621574 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:28.621472 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed9ead1f-6b8c-4823-8a20-fdda94abf5bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed9ead1f-6b8c-4823-8a20-fdda94abf5bc-maas-api-openshift-ca\") pod \"ed9ead1f-6b8c-4823-8a20-fdda94abf5bc\" (UID: \"ed9ead1f-6b8c-4823-8a20-fdda94abf5bc\") " Oct 30 19:10:28.621574 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:28.621530 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed9ead1f-6b8c-4823-8a20-fdda94abf5bc-kube-api-access-dlm6q\" (UniqueName: \"kubernetes.io/projected/ed9ead1f-6b8c-4823-8a20-fdda94abf5bc-kube-api-access-dlm6q\") pod \"ed9ead1f-6b8c-4823-8a20-fdda94abf5bc\" (UID: \"ed9ead1f-6b8c-4823-8a20-fdda94abf5bc\") " Oct 30 19:10:28.621782 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:28.621764 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed9ead1f-6b8c-4823-8a20-fdda94abf5bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed9ead1f-6b8c-4823-8a20-fdda94abf5bc-maas-api-xks-ca\") pod \"ed9ead1f-6b8c-4823-8a20-fdda94abf5bc\" (UID: \"ed9ead1f-6b8c-4823-8a20-fdda94abf5bc\") " Oct 30 19:10:28.621940 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:28.621921 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed9ead1f-6b8c-4823-8a20-fdda94abf5bc-maas-api-openshift-ca" pod "ed9ead1f-6b8c-4823-8a20-fdda94abf5bc" (UID: "ed9ead1f-6b8c-4823-8a20-fdda94abf5bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:28.622197 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:28.622180 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed9ead1f-6b8c-4823-8a20-fdda94abf5bc-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:10:28.624014 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:28.623995 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9ead1f-6b8c-4823-8a20-fdda94abf5bc-kube-api-access-dlm6q" pod "ed9ead1f-6b8c-4823-8a20-fdda94abf5bc" (UID: "ed9ead1f-6b8c-4823-8a20-fdda94abf5bc"). InnerVolumeSpecName "kube-api-access-dlm6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:28.624014 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:28.623995 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed9ead1f-6b8c-4823-8a20-fdda94abf5bc-maas-api-xks-ca" pod "ed9ead1f-6b8c-4823-8a20-fdda94abf5bc" (UID: "ed9ead1f-6b8c-4823-8a20-fdda94abf5bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:10:28.722968 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:28.722930 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed9ead1f-6b8c-4823-8a20-fdda94abf5bc-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:10:28.722968 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:28.722968 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlm6q\" (UniqueName: \"kubernetes.io/projected/ed9ead1f-6b8c-4823-8a20-fdda94abf5bc-kube-api-access-dlm6q\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:10:29.065263 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:29.065188 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4" podUID="f0df638c-bae0-471f-94a8-60e0fe66bbb5" containerName="payload-processing" containerID="cri-o://202c25f37e6421dd9475f6520a6a910eb0085b432b5d8d0f615f22b6e2f11693" gracePeriod=25 Oct 30 19:10:29.145310 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:29.145256 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w" podUID="76f80da7-3a86-4d0e-ba0b-167f66abeeba" containerName="payload-pre-processing" containerID="cri-o://1321e66ff3bacf96ecda5fcb40136a8ccbe437e39c75ad861e7cb6164f915832" gracePeriod=25 Oct 30 19:10:29.375986 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:29.375951 4531 generic.go:356] "Generic (PLEG): container finished" podID="76f80da7-3a86-4d0e-ba0b-167f66abeeba" containerID="1321e66ff3bacf96ecda5fcb40136a8ccbe437e39c75ad861e7cb6164f915832" exitCode=0 Oct 30 19:10:29.376197 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:29.376022 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w" event={"ID":"76f80da7-3a86-4d0e-ba0b-167f66abeeba","Type":"ContainerDied","Data":"1321e66ff3bacf96ecda5fcb40136a8ccbe437e39c75ad861e7cb6164f915832"} Oct 30 19:10:29.377920 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:29.377885 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782" event={"ID":"ed9ead1f-6b8c-4823-8a20-fdda94abf5bc","Type":"ContainerDied","Data":"2e9f547e6a3e3faa19919ca9252c26a0eb3c2e69890798a1be946a3eb6ae9711"} Oct 30 19:10:29.377920 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:29.377911 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782" Oct 30 19:10:29.378139 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:29.377945 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e9f547e6a3e3faa19919ca9252c26a0eb3c2e69890798a1be946a3eb6ae9711" Oct 30 19:10:29.380022 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:29.379991 4531 generic.go:356] "Generic (PLEG): container finished" podID="f0df638c-bae0-471f-94a8-60e0fe66bbb5" containerID="202c25f37e6421dd9475f6520a6a910eb0085b432b5d8d0f615f22b6e2f11693" exitCode=0 Oct 30 19:10:29.380239 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:29.380041 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4" event={"ID":"f0df638c-bae0-471f-94a8-60e0fe66bbb5","Type":"ContainerDied","Data":"202c25f37e6421dd9475f6520a6a910eb0085b432b5d8d0f615f22b6e2f11693"} Oct 30 19:10:29.405540 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:29.405506 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4" Oct 30 19:10:29.431807 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:29.431770 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0df638c-bae0-471f-94a8-60e0fe66bbb5-kube-api-access-r84sh\" (UniqueName: \"kubernetes.io/projected/f0df638c-bae0-471f-94a8-60e0fe66bbb5-kube-api-access-r84sh\") pod \"f0df638c-bae0-471f-94a8-60e0fe66bbb5\" (UID: \"f0df638c-bae0-471f-94a8-60e0fe66bbb5\") " Oct 30 19:10:29.432021 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:29.431825 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0df638c-bae0-471f-94a8-60e0fe66bbb5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0df638c-bae0-471f-94a8-60e0fe66bbb5-plugins-config-volume\") pod \"f0df638c-bae0-471f-94a8-60e0fe66bbb5\" (UID: \"f0df638c-bae0-471f-94a8-60e0fe66bbb5\") " Oct 30 19:10:29.432318 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:29.432290 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0df638c-bae0-471f-94a8-60e0fe66bbb5-plugins-config-volume" pod "f0df638c-bae0-471f-94a8-60e0fe66bbb5" (UID: "f0df638c-bae0-471f-94a8-60e0fe66bbb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:29.434766 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:29.434731 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0df638c-bae0-471f-94a8-60e0fe66bbb5-kube-api-access-r84sh" pod "f0df638c-bae0-471f-94a8-60e0fe66bbb5" (UID: "f0df638c-bae0-471f-94a8-60e0fe66bbb5"). InnerVolumeSpecName "kube-api-access-r84sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:29.462159 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:29.462097 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w" Oct 30 19:10:29.532990 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:29.532945 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76f80da7-3a86-4d0e-ba0b-167f66abeeba-kube-api-access-wvckz\" (UniqueName: \"kubernetes.io/projected/76f80da7-3a86-4d0e-ba0b-167f66abeeba-kube-api-access-wvckz\") pod \"76f80da7-3a86-4d0e-ba0b-167f66abeeba\" (UID: \"76f80da7-3a86-4d0e-ba0b-167f66abeeba\") " Oct 30 19:10:29.533451 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:29.533073 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76f80da7-3a86-4d0e-ba0b-167f66abeeba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76f80da7-3a86-4d0e-ba0b-167f66abeeba-plugins-config-volume\") pod \"76f80da7-3a86-4d0e-ba0b-167f66abeeba\" (UID: \"76f80da7-3a86-4d0e-ba0b-167f66abeeba\") " Oct 30 19:10:29.533543 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:29.533513 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76f80da7-3a86-4d0e-ba0b-167f66abeeba-plugins-config-volume" pod "76f80da7-3a86-4d0e-ba0b-167f66abeeba" (UID: "76f80da7-3a86-4d0e-ba0b-167f66abeeba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:29.533582 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:29.533552 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r84sh\" (UniqueName: \"kubernetes.io/projected/f0df638c-bae0-471f-94a8-60e0fe66bbb5-kube-api-access-r84sh\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:10:29.533582 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:29.533571 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0df638c-bae0-471f-94a8-60e0fe66bbb5-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:10:29.535418 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:29.535390 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f80da7-3a86-4d0e-ba0b-167f66abeeba-kube-api-access-wvckz" pod "76f80da7-3a86-4d0e-ba0b-167f66abeeba" (UID: "76f80da7-3a86-4d0e-ba0b-167f66abeeba"). InnerVolumeSpecName "kube-api-access-wvckz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:29.634658 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:29.634561 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvckz\" (UniqueName: \"kubernetes.io/projected/76f80da7-3a86-4d0e-ba0b-167f66abeeba-kube-api-access-wvckz\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:10:29.634658 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:29.634597 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76f80da7-3a86-4d0e-ba0b-167f66abeeba-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:10:30.387764 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:30.387660 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4" Oct 30 19:10:30.387764 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:30.387660 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4" event={"ID":"f0df638c-bae0-471f-94a8-60e0fe66bbb5","Type":"ContainerDied","Data":"f019584ba7240b9db7fbb8fec32d14d5af9f8697eef45b21b73f4b68bf131245"} Oct 30 19:10:30.388006 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:30.387816 4531 scope.go:122] "RemoveContainer" containerID="202c25f37e6421dd9475f6520a6a910eb0085b432b5d8d0f615f22b6e2f11693" Oct 30 19:10:30.389433 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:30.389404 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w" event={"ID":"76f80da7-3a86-4d0e-ba0b-167f66abeeba","Type":"ContainerDied","Data":"ca3b8cfab705942909ad1cf8ef69ad190f525b0c5c06be8b36ca0d5c05a6e224"} Oct 30 19:10:30.389562 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:30.389482 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w" Oct 30 19:10:30.398252 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:30.398091 4531 scope.go:122] "RemoveContainer" containerID="1321e66ff3bacf96ecda5fcb40136a8ccbe437e39c75ad861e7cb6164f915832" Oct 30 19:10:30.554970 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:30.554921 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4"] Oct 30 19:10:30.604986 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:30.604945 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4ebf78-cb7b974bb-tlwg4"] Oct 30 19:10:30.669721 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:30.669630 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w"] Oct 30 19:10:30.674898 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:30.674860 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4ebf78-66f848b45f-g862w"] Oct 30 19:10:32.098410 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:32.098375 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76f80da7-3a86-4d0e-ba0b-167f66abeeba" path="/var/lib/kubelet/pods/76f80da7-3a86-4d0e-ba0b-167f66abeeba/volumes" Oct 30 19:10:32.098811 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:32.098772 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0df638c-bae0-471f-94a8-60e0fe66bbb5" path="/var/lib/kubelet/pods/f0df638c-bae0-471f-94a8-60e0fe66bbb5/volumes" Oct 30 19:10:38.712905 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:38.712852 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx"] Oct 30 19:10:38.713418 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:38.713165 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx" podUID="fefa5b0d-be5a-41f5-9f0c-ac483f457148" containerName="payload-processing" containerID="cri-o://acfe7f76309ec7c235a21bc767ebe8003192114f480243b6390e7c5052896efc" gracePeriod=30 Oct 30 19:10:38.763955 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:38.763912 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh"] Oct 30 19:10:38.764232 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:38.764205 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh" podUID="6977f362-678e-4e26-bcc0-1e38c063fca5" containerName="payload-pre-processing" containerID="cri-o://15b1d05288f5eeac71cd92e83866c369432d03977e44e4d622831dff0cc4de77" gracePeriod=30 Oct 30 19:10:39.068500 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.068471 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx" Oct 30 19:10:39.072321 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.072290 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh" Oct 30 19:10:39.145724 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.145677 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fefa5b0d-be5a-41f5-9f0c-ac483f457148-kube-api-access-jhhdt\" (UniqueName: \"kubernetes.io/projected/fefa5b0d-be5a-41f5-9f0c-ac483f457148-kube-api-access-jhhdt\") pod \"fefa5b0d-be5a-41f5-9f0c-ac483f457148\" (UID: \"fefa5b0d-be5a-41f5-9f0c-ac483f457148\") " Oct 30 19:10:39.145928 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.145733 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6977f362-678e-4e26-bcc0-1e38c063fca5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6977f362-678e-4e26-bcc0-1e38c063fca5-plugins-config-volume\") pod \"6977f362-678e-4e26-bcc0-1e38c063fca5\" (UID: \"6977f362-678e-4e26-bcc0-1e38c063fca5\") " Oct 30 19:10:39.145928 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.145779 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fefa5b0d-be5a-41f5-9f0c-ac483f457148-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fefa5b0d-be5a-41f5-9f0c-ac483f457148-plugins-config-volume\") pod \"fefa5b0d-be5a-41f5-9f0c-ac483f457148\" (UID: \"fefa5b0d-be5a-41f5-9f0c-ac483f457148\") " Oct 30 19:10:39.145928 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.145811 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6977f362-678e-4e26-bcc0-1e38c063fca5-kube-api-access-fgmrs\" (UniqueName: \"kubernetes.io/projected/6977f362-678e-4e26-bcc0-1e38c063fca5-kube-api-access-fgmrs\") pod \"6977f362-678e-4e26-bcc0-1e38c063fca5\" (UID: \"6977f362-678e-4e26-bcc0-1e38c063fca5\") " Oct 30 19:10:39.146221 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.146196 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6977f362-678e-4e26-bcc0-1e38c063fca5-plugins-config-volume" pod "6977f362-678e-4e26-bcc0-1e38c063fca5" (UID: "6977f362-678e-4e26-bcc0-1e38c063fca5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:39.146298 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.146258 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fefa5b0d-be5a-41f5-9f0c-ac483f457148-plugins-config-volume" pod "fefa5b0d-be5a-41f5-9f0c-ac483f457148" (UID: "fefa5b0d-be5a-41f5-9f0c-ac483f457148"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:39.146354 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.146333 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6977f362-678e-4e26-bcc0-1e38c063fca5-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:10:39.148376 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.148346 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefa5b0d-be5a-41f5-9f0c-ac483f457148-kube-api-access-jhhdt" pod "fefa5b0d-be5a-41f5-9f0c-ac483f457148" (UID: "fefa5b0d-be5a-41f5-9f0c-ac483f457148"). InnerVolumeSpecName "kube-api-access-jhhdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:39.148439 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.148387 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6977f362-678e-4e26-bcc0-1e38c063fca5-kube-api-access-fgmrs" pod "6977f362-678e-4e26-bcc0-1e38c063fca5" (UID: "6977f362-678e-4e26-bcc0-1e38c063fca5"). InnerVolumeSpecName "kube-api-access-fgmrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:39.248404 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.248308 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fefa5b0d-be5a-41f5-9f0c-ac483f457148-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:10:39.248404 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.248353 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgmrs\" (UniqueName: \"kubernetes.io/projected/6977f362-678e-4e26-bcc0-1e38c063fca5-kube-api-access-fgmrs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:10:39.248404 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.248372 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhhdt\" (UniqueName: \"kubernetes.io/projected/fefa5b0d-be5a-41f5-9f0c-ac483f457148-kube-api-access-jhhdt\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:10:39.448008 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.447970 4531 generic.go:356] "Generic (PLEG): container finished" podID="6977f362-678e-4e26-bcc0-1e38c063fca5" containerID="15b1d05288f5eeac71cd92e83866c369432d03977e44e4d622831dff0cc4de77" exitCode=0 Oct 30 19:10:39.448353 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.448026 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh" Oct 30 19:10:39.448353 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.448040 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh" event={"ID":"6977f362-678e-4e26-bcc0-1e38c063fca5","Type":"ContainerDied","Data":"15b1d05288f5eeac71cd92e83866c369432d03977e44e4d622831dff0cc4de77"} Oct 30 19:10:39.448353 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.448096 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh" event={"ID":"6977f362-678e-4e26-bcc0-1e38c063fca5","Type":"ContainerDied","Data":"5a6a5fb9c3fd84093d67c1e58887a61493dcd55498ce5520dc3bbb95dc4c6f80"} Oct 30 19:10:39.448353 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.448109 4531 scope.go:122] "RemoveContainer" containerID="15b1d05288f5eeac71cd92e83866c369432d03977e44e4d622831dff0cc4de77" Oct 30 19:10:39.449799 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.449770 4531 generic.go:356] "Generic (PLEG): container finished" podID="fefa5b0d-be5a-41f5-9f0c-ac483f457148" containerID="acfe7f76309ec7c235a21bc767ebe8003192114f480243b6390e7c5052896efc" exitCode=0 Oct 30 19:10:39.449925 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.449832 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx" Oct 30 19:10:39.449925 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.449862 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx" event={"ID":"fefa5b0d-be5a-41f5-9f0c-ac483f457148","Type":"ContainerDied","Data":"acfe7f76309ec7c235a21bc767ebe8003192114f480243b6390e7c5052896efc"} Oct 30 19:10:39.449925 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.449914 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx" event={"ID":"fefa5b0d-be5a-41f5-9f0c-ac483f457148","Type":"ContainerDied","Data":"4ebc67ff875c702609f569e57b332f117d4e3d28397a9fc43526b6de1447aa15"} Oct 30 19:10:39.458634 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.458613 4531 scope.go:122] "RemoveContainer" containerID="15b1d05288f5eeac71cd92e83866c369432d03977e44e4d622831dff0cc4de77" Oct 30 19:10:39.459074 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:10:39.459044 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15b1d05288f5eeac71cd92e83866c369432d03977e44e4d622831dff0cc4de77\": container with ID starting with 15b1d05288f5eeac71cd92e83866c369432d03977e44e4d622831dff0cc4de77 not found: ID does not exist" containerID="15b1d05288f5eeac71cd92e83866c369432d03977e44e4d622831dff0cc4de77" Oct 30 19:10:39.459236 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.459088 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15b1d05288f5eeac71cd92e83866c369432d03977e44e4d622831dff0cc4de77"} err="failed to get container status \"15b1d05288f5eeac71cd92e83866c369432d03977e44e4d622831dff0cc4de77\": rpc error: code = NotFound desc = could not find container \"15b1d05288f5eeac71cd92e83866c369432d03977e44e4d622831dff0cc4de77\": container with ID starting with 15b1d05288f5eeac71cd92e83866c369432d03977e44e4d622831dff0cc4de77 not found: ID does not exist" Oct 30 19:10:39.459236 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.459112 4531 scope.go:122] "RemoveContainer" containerID="acfe7f76309ec7c235a21bc767ebe8003192114f480243b6390e7c5052896efc" Oct 30 19:10:39.469122 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.469096 4531 scope.go:122] "RemoveContainer" containerID="acfe7f76309ec7c235a21bc767ebe8003192114f480243b6390e7c5052896efc" Oct 30 19:10:39.469536 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:10:39.469513 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acfe7f76309ec7c235a21bc767ebe8003192114f480243b6390e7c5052896efc\": container with ID starting with acfe7f76309ec7c235a21bc767ebe8003192114f480243b6390e7c5052896efc not found: ID does not exist" containerID="acfe7f76309ec7c235a21bc767ebe8003192114f480243b6390e7c5052896efc" Oct 30 19:10:39.469604 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.469547 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acfe7f76309ec7c235a21bc767ebe8003192114f480243b6390e7c5052896efc"} err="failed to get container status \"acfe7f76309ec7c235a21bc767ebe8003192114f480243b6390e7c5052896efc\": rpc error: code = NotFound desc = could not find container \"acfe7f76309ec7c235a21bc767ebe8003192114f480243b6390e7c5052896efc\": container with ID starting with acfe7f76309ec7c235a21bc767ebe8003192114f480243b6390e7c5052896efc not found: ID does not exist" Oct 30 19:10:39.612391 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.612286 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh"] Oct 30 19:10:39.616537 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.616500 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1acf9b-6ddf64d887-g7kmh"] Oct 30 19:10:39.725316 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.725274 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx"] Oct 30 19:10:39.733319 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:39.733281 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1acf9b-789c69cb74-626rx"] Oct 30 19:10:40.098633 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:40.098592 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6977f362-678e-4e26-bcc0-1e38c063fca5" path="/var/lib/kubelet/pods/6977f362-678e-4e26-bcc0-1e38c063fca5/volumes" Oct 30 19:10:40.099002 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:40.098988 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fefa5b0d-be5a-41f5-9f0c-ac483f457148" path="/var/lib/kubelet/pods/fefa5b0d-be5a-41f5-9f0c-ac483f457148/volumes" Oct 30 19:10:40.438636 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:40.438532 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782"] Oct 30 19:10:40.442200 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:40.442167 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1acf9b-dr782"] Oct 30 19:10:42.098942 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:42.098856 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed9ead1f-6b8c-4823-8a20-fdda94abf5bc" path="/var/lib/kubelet/pods/ed9ead1f-6b8c-4823-8a20-fdda94abf5bc/volumes" Oct 30 19:10:48.611112 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.611068 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5"] Oct 30 19:10:48.611570 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.611555 4531 cpu_manager.go:414] "containerMap: removing container" podUID="6977f362-678e-4e26-bcc0-1e38c063fca5" containerName="payload-pre-processing" Oct 30 19:10:48.611621 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.611571 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6977f362-678e-4e26-bcc0-1e38c063fca5" containerName="payload-pre-processing" Oct 30 19:10:48.611621 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.611595 4531 cpu_manager.go:414] "containerMap: removing container" podUID="fefa5b0d-be5a-41f5-9f0c-ac483f457148" containerName="payload-processing" Oct 30 19:10:48.611621 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.611600 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fefa5b0d-be5a-41f5-9f0c-ac483f457148" containerName="payload-processing" Oct 30 19:10:48.611715 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.611623 4531 cpu_manager.go:414] "containerMap: removing container" podUID="ed9ead1f-6b8c-4823-8a20-fdda94abf5bc" containerName="revoke-keys" Oct 30 19:10:48.611715 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.611629 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed9ead1f-6b8c-4823-8a20-fdda94abf5bc" containerName="revoke-keys" Oct 30 19:10:48.611715 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.611636 4531 cpu_manager.go:414] "containerMap: removing container" podUID="f0df638c-bae0-471f-94a8-60e0fe66bbb5" containerName="payload-processing" Oct 30 19:10:48.611715 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.611641 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0df638c-bae0-471f-94a8-60e0fe66bbb5" containerName="payload-processing" Oct 30 19:10:48.611715 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.611652 4531 cpu_manager.go:414] "containerMap: removing container" podUID="76f80da7-3a86-4d0e-ba0b-167f66abeeba" containerName="payload-pre-processing" Oct 30 19:10:48.611715 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.611656 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76f80da7-3a86-4d0e-ba0b-167f66abeeba" containerName="payload-pre-processing" Oct 30 19:10:48.611910 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.611720 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="76f80da7-3a86-4d0e-ba0b-167f66abeeba" containerName="payload-pre-processing" Oct 30 19:10:48.611910 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.611729 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6977f362-678e-4e26-bcc0-1e38c063fca5" containerName="payload-pre-processing" Oct 30 19:10:48.611910 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.611738 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="fefa5b0d-be5a-41f5-9f0c-ac483f457148" containerName="payload-processing" Oct 30 19:10:48.611910 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.611748 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed9ead1f-6b8c-4823-8a20-fdda94abf5bc" containerName="revoke-keys" Oct 30 19:10:48.611910 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.611755 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0df638c-bae0-471f-94a8-60e0fe66bbb5" containerName="payload-processing" Oct 30 19:10:48.616423 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.616396 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5" Oct 30 19:10:48.670479 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.670444 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:10:48.670479 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.670512 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:10:48.720145 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.720092 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:10:48.720334 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.720121 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:10:48.722855 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.722821 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5"] Oct 30 19:10:48.765243 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.765198 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7lbp\" (UniqueName: \"kubernetes.io/projected/a618ecba-4d50-4dd7-941c-9e459b8fdcc6-kube-api-access-p7lbp\") pod \"maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5\" (UID: \"a618ecba-4d50-4dd7-941c-9e459b8fdcc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5" Oct 30 19:10:48.765448 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.765258 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a618ecba-4d50-4dd7-941c-9e459b8fdcc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5\" (UID: \"a618ecba-4d50-4dd7-941c-9e459b8fdcc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5" Oct 30 19:10:48.765448 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.765423 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a618ecba-4d50-4dd7-941c-9e459b8fdcc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5\" (UID: \"a618ecba-4d50-4dd7-941c-9e459b8fdcc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5" Oct 30 19:10:48.866422 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.866317 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a618ecba-4d50-4dd7-941c-9e459b8fdcc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5\" (UID: \"a618ecba-4d50-4dd7-941c-9e459b8fdcc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5" Oct 30 19:10:48.866620 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.866466 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7lbp\" (UniqueName: \"kubernetes.io/projected/a618ecba-4d50-4dd7-941c-9e459b8fdcc6-kube-api-access-p7lbp\") pod \"maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5\" (UID: \"a618ecba-4d50-4dd7-941c-9e459b8fdcc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5" Oct 30 19:10:48.866620 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.866514 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a618ecba-4d50-4dd7-941c-9e459b8fdcc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5\" (UID: \"a618ecba-4d50-4dd7-941c-9e459b8fdcc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5" Oct 30 19:10:48.867164 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.867103 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a618ecba-4d50-4dd7-941c-9e459b8fdcc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5\" (UID: \"a618ecba-4d50-4dd7-941c-9e459b8fdcc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5" Oct 30 19:10:48.869083 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.869039 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a618ecba-4d50-4dd7-941c-9e459b8fdcc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5\" (UID: \"a618ecba-4d50-4dd7-941c-9e459b8fdcc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5" Oct 30 19:10:48.928815 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.928775 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7lbp\" (UniqueName: \"kubernetes.io/projected/a618ecba-4d50-4dd7-941c-9e459b8fdcc6-kube-api-access-p7lbp\") pod \"maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5\" (UID: \"a618ecba-4d50-4dd7-941c-9e459b8fdcc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5" Oct 30 19:10:48.931631 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:48.931605 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5" Oct 30 19:10:49.310182 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:49.310119 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5"] Oct 30 19:10:49.313013 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:10:49.312980 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda618ecba_4d50_4dd7_941c_9e459b8fdcc6.slice/crio-6bbc54fd6ad036eee99f91f5cfd1c334aae6caaa481c4bd1ec974819c85a2bc1 WatchSource:0}: Error finding container 6bbc54fd6ad036eee99f91f5cfd1c334aae6caaa481c4bd1ec974819c85a2bc1: Status 404 returned error can't find the container with id 6bbc54fd6ad036eee99f91f5cfd1c334aae6caaa481c4bd1ec974819c85a2bc1 Oct 30 19:10:49.519416 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:49.519379 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5" event={"ID":"a618ecba-4d50-4dd7-941c-9e459b8fdcc6","Type":"ContainerStarted","Data":"6bbc54fd6ad036eee99f91f5cfd1c334aae6caaa481c4bd1ec974819c85a2bc1"} Oct 30 19:10:50.526882 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:50.526844 4531 generic.go:356] "Generic (PLEG): container finished" podID="a618ecba-4d50-4dd7-941c-9e459b8fdcc6" containerID="52f7a1e12879e5fe88a7f3cda027806a733d24c7ee89e7643a09858dcde40fe4" exitCode=0 Oct 30 19:10:50.527348 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:50.526929 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5" event={"ID":"a618ecba-4d50-4dd7-941c-9e459b8fdcc6","Type":"ContainerDied","Data":"52f7a1e12879e5fe88a7f3cda027806a733d24c7ee89e7643a09858dcde40fe4"} Oct 30 19:10:51.689196 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:51.689164 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5" Oct 30 19:10:51.805213 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:51.805080 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a618ecba-4d50-4dd7-941c-9e459b8fdcc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a618ecba-4d50-4dd7-941c-9e459b8fdcc6-maas-api-xks-ca\") pod \"a618ecba-4d50-4dd7-941c-9e459b8fdcc6\" (UID: \"a618ecba-4d50-4dd7-941c-9e459b8fdcc6\") " Oct 30 19:10:51.805213 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:51.805179 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a618ecba-4d50-4dd7-941c-9e459b8fdcc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a618ecba-4d50-4dd7-941c-9e459b8fdcc6-maas-api-openshift-ca\") pod \"a618ecba-4d50-4dd7-941c-9e459b8fdcc6\" (UID: \"a618ecba-4d50-4dd7-941c-9e459b8fdcc6\") " Oct 30 19:10:51.805475 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:51.805222 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a618ecba-4d50-4dd7-941c-9e459b8fdcc6-kube-api-access-p7lbp\" (UniqueName: \"kubernetes.io/projected/a618ecba-4d50-4dd7-941c-9e459b8fdcc6-kube-api-access-p7lbp\") pod \"a618ecba-4d50-4dd7-941c-9e459b8fdcc6\" (UID: \"a618ecba-4d50-4dd7-941c-9e459b8fdcc6\") " Oct 30 19:10:51.805636 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:51.805607 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a618ecba-4d50-4dd7-941c-9e459b8fdcc6-maas-api-openshift-ca" pod "a618ecba-4d50-4dd7-941c-9e459b8fdcc6" (UID: "a618ecba-4d50-4dd7-941c-9e459b8fdcc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:51.807706 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:51.807669 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a618ecba-4d50-4dd7-941c-9e459b8fdcc6-kube-api-access-p7lbp" pod "a618ecba-4d50-4dd7-941c-9e459b8fdcc6" (UID: "a618ecba-4d50-4dd7-941c-9e459b8fdcc6"). InnerVolumeSpecName "kube-api-access-p7lbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:51.807706 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:51.807680 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a618ecba-4d50-4dd7-941c-9e459b8fdcc6-maas-api-xks-ca" pod "a618ecba-4d50-4dd7-941c-9e459b8fdcc6" (UID: "a618ecba-4d50-4dd7-941c-9e459b8fdcc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:10:51.907321 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:51.907276 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7lbp\" (UniqueName: \"kubernetes.io/projected/a618ecba-4d50-4dd7-941c-9e459b8fdcc6-kube-api-access-p7lbp\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:10:51.907321 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:51.907316 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a618ecba-4d50-4dd7-941c-9e459b8fdcc6-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:10:51.907321 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:51.907331 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a618ecba-4d50-4dd7-941c-9e459b8fdcc6-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:10:52.539347 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:52.539315 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5" Oct 30 19:10:52.539570 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:52.539357 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5" event={"ID":"a618ecba-4d50-4dd7-941c-9e459b8fdcc6","Type":"ContainerDied","Data":"6bbc54fd6ad036eee99f91f5cfd1c334aae6caaa481c4bd1ec974819c85a2bc1"} Oct 30 19:10:52.539570 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:52.539423 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bbc54fd6ad036eee99f91f5cfd1c334aae6caaa481c4bd1ec974819c85a2bc1" Oct 30 19:10:54.531342 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:54.531290 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt"] Oct 30 19:10:54.655486 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:54.655450 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx"] Oct 30 19:10:56.401523 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:10:56.401483 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:10:59.531733 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:59.531679 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt" podUID="7f5e516b-073c-4d78-a0e8-5fbf82b18b93" containerName="payload-processing" containerID="cri-o://b066d6d5b5dbfc9b3e6bbec80d2b466c6345d9983768a9b8b55387bc98d8f240" gracePeriod=25 Oct 30 19:10:59.655821 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:59.655777 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx" podUID="cefd3d24-b565-4382-86cd-f408ccacf5d9" containerName="payload-pre-processing" containerID="cri-o://71370eeec09f0a0e51054d1d672be19ae08415e32ce04646bdc29a21a082c239" gracePeriod=25 Oct 30 19:10:59.818778 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:59.818737 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt" Oct 30 19:10:59.894516 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:59.894483 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f5e516b-073c-4d78-a0e8-5fbf82b18b93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f5e516b-073c-4d78-a0e8-5fbf82b18b93-plugins-config-volume\") pod \"7f5e516b-073c-4d78-a0e8-5fbf82b18b93\" (UID: \"7f5e516b-073c-4d78-a0e8-5fbf82b18b93\") " Oct 30 19:10:59.894711 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:59.894693 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f5e516b-073c-4d78-a0e8-5fbf82b18b93-kube-api-access-l9slt\" (UniqueName: \"kubernetes.io/projected/7f5e516b-073c-4d78-a0e8-5fbf82b18b93-kube-api-access-l9slt\") pod \"7f5e516b-073c-4d78-a0e8-5fbf82b18b93\" (UID: \"7f5e516b-073c-4d78-a0e8-5fbf82b18b93\") " Oct 30 19:10:59.894979 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:59.894952 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f5e516b-073c-4d78-a0e8-5fbf82b18b93-plugins-config-volume" pod "7f5e516b-073c-4d78-a0e8-5fbf82b18b93" (UID: "7f5e516b-073c-4d78-a0e8-5fbf82b18b93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:59.895112 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:59.895086 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f5e516b-073c-4d78-a0e8-5fbf82b18b93-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:10:59.897094 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:59.897061 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5e516b-073c-4d78-a0e8-5fbf82b18b93-kube-api-access-l9slt" pod "7f5e516b-073c-4d78-a0e8-5fbf82b18b93" (UID: "7f5e516b-073c-4d78-a0e8-5fbf82b18b93"). InnerVolumeSpecName "kube-api-access-l9slt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:59.915425 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:59.915395 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx" Oct 30 19:10:59.995669 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:59.995628 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cefd3d24-b565-4382-86cd-f408ccacf5d9-kube-api-access-4v6cc\" (UniqueName: \"kubernetes.io/projected/cefd3d24-b565-4382-86cd-f408ccacf5d9-kube-api-access-4v6cc\") pod \"cefd3d24-b565-4382-86cd-f408ccacf5d9\" (UID: \"cefd3d24-b565-4382-86cd-f408ccacf5d9\") " Oct 30 19:10:59.995873 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:59.995678 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cefd3d24-b565-4382-86cd-f408ccacf5d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cefd3d24-b565-4382-86cd-f408ccacf5d9-plugins-config-volume\") pod \"cefd3d24-b565-4382-86cd-f408ccacf5d9\" (UID: \"cefd3d24-b565-4382-86cd-f408ccacf5d9\") " Oct 30 19:10:59.996236 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:59.996211 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9slt\" (UniqueName: \"kubernetes.io/projected/7f5e516b-073c-4d78-a0e8-5fbf82b18b93-kube-api-access-l9slt\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:10:59.996343 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:59.996233 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cefd3d24-b565-4382-86cd-f408ccacf5d9-plugins-config-volume" pod "cefd3d24-b565-4382-86cd-f408ccacf5d9" (UID: "cefd3d24-b565-4382-86cd-f408ccacf5d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:59.998289 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:10:59.998259 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cefd3d24-b565-4382-86cd-f408ccacf5d9-kube-api-access-4v6cc" pod "cefd3d24-b565-4382-86cd-f408ccacf5d9" (UID: "cefd3d24-b565-4382-86cd-f408ccacf5d9"). InnerVolumeSpecName "kube-api-access-4v6cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:11:00.097204 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:00.097093 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v6cc\" (UniqueName: \"kubernetes.io/projected/cefd3d24-b565-4382-86cd-f408ccacf5d9-kube-api-access-4v6cc\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:11:00.097204 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:00.097156 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cefd3d24-b565-4382-86cd-f408ccacf5d9-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:11:00.588919 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:00.588881 4531 generic.go:356] "Generic (PLEG): container finished" podID="7f5e516b-073c-4d78-a0e8-5fbf82b18b93" containerID="b066d6d5b5dbfc9b3e6bbec80d2b466c6345d9983768a9b8b55387bc98d8f240" exitCode=0 Oct 30 19:11:00.589437 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:00.588935 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt" Oct 30 19:11:00.589437 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:00.588971 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt" event={"ID":"7f5e516b-073c-4d78-a0e8-5fbf82b18b93","Type":"ContainerDied","Data":"b066d6d5b5dbfc9b3e6bbec80d2b466c6345d9983768a9b8b55387bc98d8f240"} Oct 30 19:11:00.589437 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:00.589031 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt" event={"ID":"7f5e516b-073c-4d78-a0e8-5fbf82b18b93","Type":"ContainerDied","Data":"c7fbfd0da0dcd94be1d378e96eaa664715cc308ee66a22ade468bc39153370e3"} Oct 30 19:11:00.589437 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:00.589055 4531 scope.go:122] "RemoveContainer" containerID="b066d6d5b5dbfc9b3e6bbec80d2b466c6345d9983768a9b8b55387bc98d8f240" Oct 30 19:11:00.590870 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:00.590751 4531 generic.go:356] "Generic (PLEG): container finished" podID="cefd3d24-b565-4382-86cd-f408ccacf5d9" containerID="71370eeec09f0a0e51054d1d672be19ae08415e32ce04646bdc29a21a082c239" exitCode=0 Oct 30 19:11:00.590870 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:00.590809 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx" Oct 30 19:11:00.590870 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:00.590843 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx" event={"ID":"cefd3d24-b565-4382-86cd-f408ccacf5d9","Type":"ContainerDied","Data":"71370eeec09f0a0e51054d1d672be19ae08415e32ce04646bdc29a21a082c239"} Oct 30 19:11:00.591120 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:00.590907 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx" event={"ID":"cefd3d24-b565-4382-86cd-f408ccacf5d9","Type":"ContainerDied","Data":"3d4b125bc36e693cfdaf56575dcd74cfa1e13c4d938cfc7816db71e6a9563bea"} Oct 30 19:11:00.600824 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:00.600801 4531 scope.go:122] "RemoveContainer" containerID="b066d6d5b5dbfc9b3e6bbec80d2b466c6345d9983768a9b8b55387bc98d8f240" Oct 30 19:11:00.601222 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:11:00.601194 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b066d6d5b5dbfc9b3e6bbec80d2b466c6345d9983768a9b8b55387bc98d8f240\": container with ID starting with b066d6d5b5dbfc9b3e6bbec80d2b466c6345d9983768a9b8b55387bc98d8f240 not found: ID does not exist" containerID="b066d6d5b5dbfc9b3e6bbec80d2b466c6345d9983768a9b8b55387bc98d8f240" Oct 30 19:11:00.601344 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:00.601229 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b066d6d5b5dbfc9b3e6bbec80d2b466c6345d9983768a9b8b55387bc98d8f240"} err="failed to get container status \"b066d6d5b5dbfc9b3e6bbec80d2b466c6345d9983768a9b8b55387bc98d8f240\": rpc error: code = NotFound desc = could not find container \"b066d6d5b5dbfc9b3e6bbec80d2b466c6345d9983768a9b8b55387bc98d8f240\": container with ID starting with b066d6d5b5dbfc9b3e6bbec80d2b466c6345d9983768a9b8b55387bc98d8f240 not found: ID does not exist" Oct 30 19:11:00.601344 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:00.601247 4531 scope.go:122] "RemoveContainer" containerID="71370eeec09f0a0e51054d1d672be19ae08415e32ce04646bdc29a21a082c239" Oct 30 19:11:00.611083 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:00.611054 4531 scope.go:122] "RemoveContainer" containerID="71370eeec09f0a0e51054d1d672be19ae08415e32ce04646bdc29a21a082c239" Oct 30 19:11:00.611450 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:11:00.611429 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71370eeec09f0a0e51054d1d672be19ae08415e32ce04646bdc29a21a082c239\": container with ID starting with 71370eeec09f0a0e51054d1d672be19ae08415e32ce04646bdc29a21a082c239 not found: ID does not exist" containerID="71370eeec09f0a0e51054d1d672be19ae08415e32ce04646bdc29a21a082c239" Oct 30 19:11:00.611513 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:00.611464 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71370eeec09f0a0e51054d1d672be19ae08415e32ce04646bdc29a21a082c239"} err="failed to get container status \"71370eeec09f0a0e51054d1d672be19ae08415e32ce04646bdc29a21a082c239\": rpc error: code = NotFound desc = could not find container \"71370eeec09f0a0e51054d1d672be19ae08415e32ce04646bdc29a21a082c239\": container with ID starting with 71370eeec09f0a0e51054d1d672be19ae08415e32ce04646bdc29a21a082c239 not found: ID does not exist" Oct 30 19:11:00.758407 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:00.758368 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx"] Oct 30 19:11:00.767974 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:00.767944 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-6f4bb6897f-vswtx"] Oct 30 19:11:00.879478 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:00.879384 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt"] Oct 30 19:11:00.883084 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:00.883046 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-6c5f445564-dnmpt"] Oct 30 19:11:01.907424 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:01.907389 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6"] Oct 30 19:11:01.907853 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:01.907840 4531 cpu_manager.go:414] "containerMap: removing container" podUID="7f5e516b-073c-4d78-a0e8-5fbf82b18b93" containerName="payload-processing" Oct 30 19:11:01.907901 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:01.907854 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f5e516b-073c-4d78-a0e8-5fbf82b18b93" containerName="payload-processing" Oct 30 19:11:01.907901 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:01.907866 4531 cpu_manager.go:414] "containerMap: removing container" podUID="a618ecba-4d50-4dd7-941c-9e459b8fdcc6" containerName="revoke-keys" Oct 30 19:11:01.907901 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:01.907870 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a618ecba-4d50-4dd7-941c-9e459b8fdcc6" containerName="revoke-keys" Oct 30 19:11:01.907901 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:01.907885 4531 cpu_manager.go:414] "containerMap: removing container" podUID="cefd3d24-b565-4382-86cd-f408ccacf5d9" containerName="payload-pre-processing" Oct 30 19:11:01.907901 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:01.907893 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cefd3d24-b565-4382-86cd-f408ccacf5d9" containerName="payload-pre-processing" Oct 30 19:11:01.908056 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:01.907969 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="cefd3d24-b565-4382-86cd-f408ccacf5d9" containerName="payload-pre-processing" Oct 30 19:11:01.908056 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:01.907982 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f5e516b-073c-4d78-a0e8-5fbf82b18b93" containerName="payload-processing" Oct 30 19:11:01.908056 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:01.907990 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a618ecba-4d50-4dd7-941c-9e459b8fdcc6" containerName="revoke-keys" Oct 30 19:11:01.912626 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:01.912604 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" Oct 30 19:11:01.964915 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:01.964882 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-0c9a2b\"/\"openshift-service-ca.crt\"" Oct 30 19:11:02.014192 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:02.014156 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-0c9a2b\"/\"kube-root-ca.crt\"" Oct 30 19:11:02.014384 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:02.014293 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-0c9a2b\"/\"ipp-route-0c9a2b-kserve-self-signed-certs\"" Oct 30 19:11:02.014384 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:02.014350 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-0c9a2b\"/\"default-dockercfg-pzr2h\"" Oct 30 19:11:02.016912 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:02.016878 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6"] Oct 30 19:11:02.019848 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:02.019816 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-home\") pod \"ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6\" (UID: \"6796af55-3e88-4dec-9b8a-9232d8c10e16\") " pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" Oct 30 19:11:02.019973 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:02.019867 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv4gp\" (UniqueName: \"kubernetes.io/projected/6796af55-3e88-4dec-9b8a-9232d8c10e16-kube-api-access-nv4gp\") pod \"ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6\" (UID: \"6796af55-3e88-4dec-9b8a-9232d8c10e16\") " pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" Oct 30 19:11:02.020035 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:02.019996 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-tmp-dir\") pod \"ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6\" (UID: \"6796af55-3e88-4dec-9b8a-9232d8c10e16\") " pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" Oct 30 19:11:02.020082 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:02.020056 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-model-cache\") pod \"ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6\" (UID: \"6796af55-3e88-4dec-9b8a-9232d8c10e16\") " pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" Oct 30 19:11:02.020196 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:02.020179 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6796af55-3e88-4dec-9b8a-9232d8c10e16-tls-certs\") pod \"ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6\" (UID: \"6796af55-3e88-4dec-9b8a-9232d8c10e16\") " pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" Oct 30 19:11:02.020282 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:02.020269 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-dshm\") pod \"ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6\" (UID: \"6796af55-3e88-4dec-9b8a-9232d8c10e16\") " pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" Oct 30 19:11:02.099719 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:02.099678 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f5e516b-073c-4d78-a0e8-5fbf82b18b93" path="/var/lib/kubelet/pods/7f5e516b-073c-4d78-a0e8-5fbf82b18b93/volumes" Oct 30 19:11:02.100171 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:02.100119 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cefd3d24-b565-4382-86cd-f408ccacf5d9" path="/var/lib/kubelet/pods/cefd3d24-b565-4382-86cd-f408ccacf5d9/volumes" Oct 30 19:11:02.121017 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:02.120974 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-dshm\") pod \"ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6\" (UID: \"6796af55-3e88-4dec-9b8a-9232d8c10e16\") " pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" Oct 30 19:11:02.121280 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:02.121077 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-home\") pod \"ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6\" (UID: \"6796af55-3e88-4dec-9b8a-9232d8c10e16\") " pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" Oct 30 19:11:02.121280 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:02.121116 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv4gp\" (UniqueName: \"kubernetes.io/projected/6796af55-3e88-4dec-9b8a-9232d8c10e16-kube-api-access-nv4gp\") pod \"ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6\" (UID: \"6796af55-3e88-4dec-9b8a-9232d8c10e16\") " pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" Oct 30 19:11:02.121280 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:02.121234 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-tmp-dir\") pod \"ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6\" (UID: \"6796af55-3e88-4dec-9b8a-9232d8c10e16\") " pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" Oct 30 19:11:02.121488 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:02.121305 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-model-cache\") pod \"ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6\" (UID: \"6796af55-3e88-4dec-9b8a-9232d8c10e16\") " pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" Oct 30 19:11:02.121488 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:02.121349 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6796af55-3e88-4dec-9b8a-9232d8c10e16-tls-certs\") pod \"ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6\" (UID: \"6796af55-3e88-4dec-9b8a-9232d8c10e16\") " pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" Oct 30 19:11:02.121693 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:02.121665 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-tmp-dir\") pod \"ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6\" (UID: \"6796af55-3e88-4dec-9b8a-9232d8c10e16\") " pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" Oct 30 19:11:02.121856 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:02.121836 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-model-cache\") pod \"ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6\" (UID: \"6796af55-3e88-4dec-9b8a-9232d8c10e16\") " pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" Oct 30 19:11:02.122240 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:02.122216 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-home\") pod \"ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6\" (UID: \"6796af55-3e88-4dec-9b8a-9232d8c10e16\") " pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" Oct 30 19:11:02.123718 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:02.123688 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-dshm\") pod \"ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6\" (UID: \"6796af55-3e88-4dec-9b8a-9232d8c10e16\") " pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" Oct 30 19:11:02.124196 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:02.124122 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6796af55-3e88-4dec-9b8a-9232d8c10e16-tls-certs\") pod \"ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6\" (UID: \"6796af55-3e88-4dec-9b8a-9232d8c10e16\") " pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" Oct 30 19:11:02.183652 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:02.183611 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv4gp\" (UniqueName: \"kubernetes.io/projected/6796af55-3e88-4dec-9b8a-9232d8c10e16-kube-api-access-nv4gp\") pod \"ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6\" (UID: \"6796af55-3e88-4dec-9b8a-9232d8c10e16\") " pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" Oct 30 19:11:02.226145 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:02.226069 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" Oct 30 19:11:02.621980 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:02.621900 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6"] Oct 30 19:11:02.624668 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:11:02.624624 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6796af55_3e88_4dec_9b8a_9232d8c10e16.slice/crio-a444ab54a8653fc70746951da64c589324e197e03456e80ac14e04a80f0dced9 WatchSource:0}: Error finding container a444ab54a8653fc70746951da64c589324e197e03456e80ac14e04a80f0dced9: Status 404 returned error can't find the container with id a444ab54a8653fc70746951da64c589324e197e03456e80ac14e04a80f0dced9 Oct 30 19:11:03.613875 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:03.613835 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" event={"ID":"6796af55-3e88-4dec-9b8a-9232d8c10e16","Type":"ContainerStarted","Data":"156ba02f779d4dbc5f228b816a66fe061680bc56db81bb5c0e206c5b513d18d1"} Oct 30 19:11:03.614330 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:03.613889 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" event={"ID":"6796af55-3e88-4dec-9b8a-9232d8c10e16","Type":"ContainerStarted","Data":"a444ab54a8653fc70746951da64c589324e197e03456e80ac14e04a80f0dced9"} Oct 30 19:11:03.734006 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:03.733929 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" podStartSLOduration=2.733908083 podStartE2EDuration="2.733908083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:03.731687103 +0000 UTC m=+2223.758792063" watchObservedRunningTime="2026-10-30 19:11:03.733908083 +0000 UTC m=+2223.761013047" Oct 30 19:11:04.227164 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:04.227087 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" Oct 30 19:11:12.227168 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:12.227091 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" Oct 30 19:11:12.240812 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:12.240783 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" Oct 30 19:11:12.680637 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:12.680603 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" Oct 30 19:11:21.624904 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:21.624861 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5"] Oct 30 19:11:21.673145 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:21.673092 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4ebf78-mgxl5"] Oct 30 19:11:22.099021 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:22.098987 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a618ecba-4d50-4dd7-941c-9e459b8fdcc6" path="/var/lib/kubelet/pods/a618ecba-4d50-4dd7-941c-9e459b8fdcc6/volumes" Oct 30 19:11:26.425996 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:11:26.425949 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:11:28.030351 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:28.030309 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67448-bc4nw"] Oct 30 19:11:28.078320 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:28.078284 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67448-bc4nw"] Oct 30 19:11:28.098741 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:28.098705 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="989b7485-a99c-4dae-af11-851e6fd6205b" path="/var/lib/kubelet/pods/989b7485-a99c-4dae-af11-851e6fd6205b/volumes" Oct 30 19:11:37.835637 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:37.835589 4531 generic.go:356] "Generic (PLEG): container finished" podID="cac0c9e8-98a5-49b1-93c4-944bfda920a2" containerID="132f38aa4f207a572b9c55dbf6123e70927c6484a46538a435913fec0cb0b204" exitCode=0 Oct 30 19:11:37.836187 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:37.835665 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" event={"ID":"cac0c9e8-98a5-49b1-93c4-944bfda920a2","Type":"ContainerDied","Data":"132f38aa4f207a572b9c55dbf6123e70927c6484a46538a435913fec0cb0b204"} Oct 30 19:11:37.836254 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:37.836239 4531 scope.go:122] "RemoveContainer" containerID="132f38aa4f207a572b9c55dbf6123e70927c6484a46538a435913fec0cb0b204" Oct 30 19:11:37.838091 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:37.838055 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:11:37.838219 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:37.838163 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:11:37.838219 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:37.838209 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:11:38.846783 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:38.846737 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" event={"ID":"cac0c9e8-98a5-49b1-93c4-944bfda920a2","Type":"ContainerStarted","Data":"c6f6e39e85a67bdf500076d74258c06679610567828a33329ce5a2f251ca9030"} Oct 30 19:11:39.191282 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:39.191180 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:11:39.191496 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:39.191472 4531 patch_prober.go:29] interesting pod/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 19:11:39.191584 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:39.191551 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" podUID="cac0c9e8-98a5-49b1-93c4-944bfda920a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 19:11:40.191843 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:40.191804 4531 patch_prober.go:29] interesting pod/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 19:11:40.192337 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:40.191864 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" podUID="cac0c9e8-98a5-49b1-93c4-944bfda920a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 19:11:40.694231 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:40.694169 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw"] Oct 30 19:11:40.698274 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:40.698241 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw" Oct 30 19:11:40.753511 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:40.753475 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-f554bb\"" Oct 30 19:11:40.755972 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:40.755018 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-f554bb-dockercfg-vqk9l\"" Oct 30 19:11:40.805395 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:40.805345 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw"] Oct 30 19:11:40.856552 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:40.856518 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d03a78-d49d-45ee-a7f4-cc38e6e651cd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw\" (UID: \"84d03a78-d49d-45ee-a7f4-cc38e6e651cd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw" Oct 30 19:11:40.856756 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:40.856658 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxsth\" (UniqueName: \"kubernetes.io/projected/84d03a78-d49d-45ee-a7f4-cc38e6e651cd-kube-api-access-cxsth\") pod \"payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw\" (UID: \"84d03a78-d49d-45ee-a7f4-cc38e6e651cd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw" Oct 30 19:11:40.958430 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:40.958321 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d03a78-d49d-45ee-a7f4-cc38e6e651cd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw\" (UID: \"84d03a78-d49d-45ee-a7f4-cc38e6e651cd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw" Oct 30 19:11:40.958623 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:40.958441 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxsth\" (UniqueName: \"kubernetes.io/projected/84d03a78-d49d-45ee-a7f4-cc38e6e651cd-kube-api-access-cxsth\") pod \"payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw\" (UID: \"84d03a78-d49d-45ee-a7f4-cc38e6e651cd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw" Oct 30 19:11:40.959122 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:40.959095 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d03a78-d49d-45ee-a7f4-cc38e6e651cd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw\" (UID: \"84d03a78-d49d-45ee-a7f4-cc38e6e651cd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw" Oct 30 19:11:41.042704 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:41.042671 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxsth\" (UniqueName: \"kubernetes.io/projected/84d03a78-d49d-45ee-a7f4-cc38e6e651cd-kube-api-access-cxsth\") pod \"payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw\" (UID: \"84d03a78-d49d-45ee-a7f4-cc38e6e651cd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw" Oct 30 19:11:41.191900 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:41.191862 4531 patch_prober.go:29] interesting pod/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 19:11:41.192366 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:41.191917 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" podUID="cac0c9e8-98a5-49b1-93c4-944bfda920a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 19:11:41.355157 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:41.344358 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw" Oct 30 19:11:41.760445 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:41.760413 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw"] Oct 30 19:11:41.762708 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:11:41.762665 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d03a78_d49d_45ee_a7f4_cc38e6e651cd.slice/crio-b0f9241660d0ea0c095a36f6c9bf13a1cd01a3443dd2b78b71619a50a6e8c46a WatchSource:0}: Error finding container b0f9241660d0ea0c095a36f6c9bf13a1cd01a3443dd2b78b71619a50a6e8c46a: Status 404 returned error can't find the container with id b0f9241660d0ea0c095a36f6c9bf13a1cd01a3443dd2b78b71619a50a6e8c46a Oct 30 19:11:41.871090 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:41.871055 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw" event={"ID":"84d03a78-d49d-45ee-a7f4-cc38e6e651cd","Type":"ContainerStarted","Data":"993bd5d58ddd2d94d4251f46dd6515318b84632628cddd3a129b496e510afb19"} Oct 30 19:11:41.871307 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:41.871109 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw" event={"ID":"84d03a78-d49d-45ee-a7f4-cc38e6e651cd","Type":"ContainerStarted","Data":"b0f9241660d0ea0c095a36f6c9bf13a1cd01a3443dd2b78b71619a50a6e8c46a"} Oct 30 19:11:41.871307 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:41.871206 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw" Oct 30 19:11:42.002753 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:42.002692 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw" podStartSLOduration=2.002676195 podStartE2EDuration="2.002676195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:42.001581195 +0000 UTC m=+2262.028686153" watchObservedRunningTime="2026-10-30 19:11:42.002676195 +0000 UTC m=+2262.029781153" Oct 30 19:11:42.191782 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:42.191699 4531 patch_prober.go:29] interesting pod/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 19:11:42.191782 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:42.191760 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" podUID="cac0c9e8-98a5-49b1-93c4-944bfda920a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 19:11:43.192244 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:43.192200 4531 patch_prober.go:29] interesting pod/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 19:11:43.192760 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:43.192269 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" podUID="cac0c9e8-98a5-49b1-93c4-944bfda920a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 19:11:44.183263 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:44.183214 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:11:44.281465 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:44.281320 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:11:44.890461 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:44.890430 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:11:51.309012 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:51.308971 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs"] Oct 30 19:11:51.311879 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:51.311857 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs" Oct 30 19:11:51.364291 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:51.364259 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7f44c7-dockercfg-rhnlr\"" Oct 30 19:11:51.386153 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:51.386081 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0677eeba-3e54-4b81-a294-38435cc77ef9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs\" (UID: \"0677eeba-3e54-4b81-a294-38435cc77ef9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs" Oct 30 19:11:51.386351 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:51.386221 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm844\" (UniqueName: \"kubernetes.io/projected/0677eeba-3e54-4b81-a294-38435cc77ef9-kube-api-access-rm844\") pod \"payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs\" (UID: \"0677eeba-3e54-4b81-a294-38435cc77ef9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs" Oct 30 19:11:51.413608 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:51.413570 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7f44c7\"" Oct 30 19:11:51.416062 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:51.416037 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs"] Oct 30 19:11:51.487720 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:51.487686 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0677eeba-3e54-4b81-a294-38435cc77ef9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs\" (UID: \"0677eeba-3e54-4b81-a294-38435cc77ef9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs" Oct 30 19:11:51.487883 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:51.487725 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm844\" (UniqueName: \"kubernetes.io/projected/0677eeba-3e54-4b81-a294-38435cc77ef9-kube-api-access-rm844\") pod \"payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs\" (UID: \"0677eeba-3e54-4b81-a294-38435cc77ef9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs" Oct 30 19:11:51.488459 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:51.488439 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0677eeba-3e54-4b81-a294-38435cc77ef9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs\" (UID: \"0677eeba-3e54-4b81-a294-38435cc77ef9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs" Oct 30 19:11:51.548905 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:51.548869 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm844\" (UniqueName: \"kubernetes.io/projected/0677eeba-3e54-4b81-a294-38435cc77ef9-kube-api-access-rm844\") pod \"payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs\" (UID: \"0677eeba-3e54-4b81-a294-38435cc77ef9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs" Oct 30 19:11:51.625428 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:51.625335 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs" Oct 30 19:11:52.001294 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:52.001266 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs"] Oct 30 19:11:52.003301 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:11:52.003271 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0677eeba_3e54_4b81_a294_38435cc77ef9.slice/crio-8efe50238b53e3f26367d23a0916631f0d8390a588f906b6f96c502d0bf839fd WatchSource:0}: Error finding container 8efe50238b53e3f26367d23a0916631f0d8390a588f906b6f96c502d0bf839fd: Status 404 returned error can't find the container with id 8efe50238b53e3f26367d23a0916631f0d8390a588f906b6f96c502d0bf839fd Oct 30 19:11:52.110526 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:52.110493 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62"] Oct 30 19:11:52.113096 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:52.113074 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62" Oct 30 19:11:52.174286 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:52.174250 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62"] Oct 30 19:11:52.197864 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:52.197821 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt8lw\" (UniqueName: \"kubernetes.io/projected/5bbecb8e-40ce-4b0d-948f-c7b53598da3f-kube-api-access-gt8lw\") pod \"payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62\" (UID: \"5bbecb8e-40ce-4b0d-948f-c7b53598da3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62" Oct 30 19:11:52.198080 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:52.197969 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bbecb8e-40ce-4b0d-948f-c7b53598da3f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62\" (UID: \"5bbecb8e-40ce-4b0d-948f-c7b53598da3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62" Oct 30 19:11:52.299101 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:52.298988 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt8lw\" (UniqueName: \"kubernetes.io/projected/5bbecb8e-40ce-4b0d-948f-c7b53598da3f-kube-api-access-gt8lw\") pod \"payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62\" (UID: \"5bbecb8e-40ce-4b0d-948f-c7b53598da3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62" Oct 30 19:11:52.299101 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:52.299062 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bbecb8e-40ce-4b0d-948f-c7b53598da3f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62\" (UID: \"5bbecb8e-40ce-4b0d-948f-c7b53598da3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62" Oct 30 19:11:52.299883 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:52.299858 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bbecb8e-40ce-4b0d-948f-c7b53598da3f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62\" (UID: \"5bbecb8e-40ce-4b0d-948f-c7b53598da3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62" Oct 30 19:11:52.360512 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:52.360481 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt8lw\" (UniqueName: \"kubernetes.io/projected/5bbecb8e-40ce-4b0d-948f-c7b53598da3f-kube-api-access-gt8lw\") pod \"payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62\" (UID: \"5bbecb8e-40ce-4b0d-948f-c7b53598da3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62" Oct 30 19:11:52.426807 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:52.426772 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62" Oct 30 19:11:52.804019 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:52.803583 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62"] Oct 30 19:11:52.806699 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:11:52.806662 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bbecb8e_40ce_4b0d_948f_c7b53598da3f.slice/crio-fbc874a82ed8da0a7748e8ae21a6e00d0e21091095dc3ba2633d349827bc38c8 WatchSource:0}: Error finding container fbc874a82ed8da0a7748e8ae21a6e00d0e21091095dc3ba2633d349827bc38c8: Status 404 returned error can't find the container with id fbc874a82ed8da0a7748e8ae21a6e00d0e21091095dc3ba2633d349827bc38c8 Oct 30 19:11:52.880941 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:52.880906 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw" Oct 30 19:11:52.947563 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:52.947522 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62" event={"ID":"5bbecb8e-40ce-4b0d-948f-c7b53598da3f","Type":"ContainerStarted","Data":"5298e821c3f4ff6dc1e9ac0d9e8830833ac86291d9468266d2eaf04340b22ad1"} Oct 30 19:11:52.947774 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:52.947593 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62" event={"ID":"5bbecb8e-40ce-4b0d-948f-c7b53598da3f","Type":"ContainerStarted","Data":"fbc874a82ed8da0a7748e8ae21a6e00d0e21091095dc3ba2633d349827bc38c8"} Oct 30 19:11:52.947774 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:52.947624 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62" Oct 30 19:11:52.949453 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:52.949418 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs" event={"ID":"0677eeba-3e54-4b81-a294-38435cc77ef9","Type":"ContainerStarted","Data":"dcbd49ddbc3af98ba7ea895af733ce704522d2d7d988e31bece622b0bf6a7e6c"} Oct 30 19:11:52.949627 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:52.949481 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs" event={"ID":"0677eeba-3e54-4b81-a294-38435cc77ef9","Type":"ContainerStarted","Data":"8efe50238b53e3f26367d23a0916631f0d8390a588f906b6f96c502d0bf839fd"} Oct 30 19:11:52.949627 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:52.949534 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs" Oct 30 19:11:53.118431 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:11:53.118318 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62" podStartSLOduration=1.118300639 podStartE2EDuration="1.118300639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:53.115899145 +0000 UTC m=+2273.143004104" watchObservedRunningTime="2026-10-30 19:11:53.118300639 +0000 UTC m=+2273.145405597" Oct 30 19:11:56.396925 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:11:56.396873 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:12:03.957592 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:03.957543 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62" Oct 30 19:12:03.958181 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:03.957672 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs" Oct 30 19:12:04.089329 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:04.089258 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs" podStartSLOduration=13.089237556 podStartE2EDuration="13.089237556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:53.241384136 +0000 UTC m=+2273.268489091" watchObservedRunningTime="2026-10-30 19:12:04.089237556 +0000 UTC m=+2284.116342516" Oct 30 19:12:13.923939 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:13.923849 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj"] Oct 30 19:12:13.929183 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:13.929161 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj" Oct 30 19:12:13.944932 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:13.944893 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n4hq\" (UniqueName: \"kubernetes.io/projected/a80bd19d-b0c9-48f8-a200-2c59d57c2ed1-kube-api-access-5n4hq\") pod \"maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj\" (UID: \"a80bd19d-b0c9-48f8-a200-2c59d57c2ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj" Oct 30 19:12:13.945093 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:13.944988 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a80bd19d-b0c9-48f8-a200-2c59d57c2ed1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj\" (UID: \"a80bd19d-b0c9-48f8-a200-2c59d57c2ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj" Oct 30 19:12:13.945216 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:13.945199 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a80bd19d-b0c9-48f8-a200-2c59d57c2ed1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj\" (UID: \"a80bd19d-b0c9-48f8-a200-2c59d57c2ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj" Oct 30 19:12:13.988193 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:13.988156 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj"] Oct 30 19:12:14.033552 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:14.033514 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:12:14.033751 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:14.033681 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:12:14.035750 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:14.035721 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:12:14.036066 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:14.036042 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:12:14.046111 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:14.046064 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a80bd19d-b0c9-48f8-a200-2c59d57c2ed1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj\" (UID: \"a80bd19d-b0c9-48f8-a200-2c59d57c2ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj" Oct 30 19:12:14.046375 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:14.046193 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n4hq\" (UniqueName: \"kubernetes.io/projected/a80bd19d-b0c9-48f8-a200-2c59d57c2ed1-kube-api-access-5n4hq\") pod \"maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj\" (UID: \"a80bd19d-b0c9-48f8-a200-2c59d57c2ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj" Oct 30 19:12:14.046375 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:14.046240 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a80bd19d-b0c9-48f8-a200-2c59d57c2ed1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj\" (UID: \"a80bd19d-b0c9-48f8-a200-2c59d57c2ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj" Oct 30 19:12:14.046939 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:14.046913 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a80bd19d-b0c9-48f8-a200-2c59d57c2ed1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj\" (UID: \"a80bd19d-b0c9-48f8-a200-2c59d57c2ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj" Oct 30 19:12:14.049217 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:14.049184 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a80bd19d-b0c9-48f8-a200-2c59d57c2ed1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj\" (UID: \"a80bd19d-b0c9-48f8-a200-2c59d57c2ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj" Oct 30 19:12:14.123287 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:14.123252 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n4hq\" (UniqueName: \"kubernetes.io/projected/a80bd19d-b0c9-48f8-a200-2c59d57c2ed1-kube-api-access-5n4hq\") pod \"maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj\" (UID: \"a80bd19d-b0c9-48f8-a200-2c59d57c2ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj" Oct 30 19:12:14.242799 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:14.242758 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj" Oct 30 19:12:14.621898 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:14.621795 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj"] Oct 30 19:12:14.625012 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:12:14.624977 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80bd19d_b0c9_48f8_a200_2c59d57c2ed1.slice/crio-09db5db0d8e05875661278e0ffc40d3f1a9f4436a7735594055acdaf9acfd396 WatchSource:0}: Error finding container 09db5db0d8e05875661278e0ffc40d3f1a9f4436a7735594055acdaf9acfd396: Status 404 returned error can't find the container with id 09db5db0d8e05875661278e0ffc40d3f1a9f4436a7735594055acdaf9acfd396 Oct 30 19:12:15.130094 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:15.130055 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj" event={"ID":"a80bd19d-b0c9-48f8-a200-2c59d57c2ed1","Type":"ContainerStarted","Data":"dcebb7b1348d6f7eaf2e71c2e341576b2186f15c07b2e29057820e210c2743f7"} Oct 30 19:12:15.130590 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:15.130115 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj" event={"ID":"a80bd19d-b0c9-48f8-a200-2c59d57c2ed1","Type":"ContainerStarted","Data":"09db5db0d8e05875661278e0ffc40d3f1a9f4436a7735594055acdaf9acfd396"} Oct 30 19:12:15.247187 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:15.247103 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj" podStartSLOduration=2.247087888 podStartE2EDuration="2.247087888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:15.245608299 +0000 UTC m=+2295.272713260" watchObservedRunningTime="2026-10-30 19:12:15.247087888 +0000 UTC m=+2295.274192846" Oct 30 19:12:16.139002 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:16.138963 4531 generic.go:356] "Generic (PLEG): container finished" podID="a80bd19d-b0c9-48f8-a200-2c59d57c2ed1" containerID="dcebb7b1348d6f7eaf2e71c2e341576b2186f15c07b2e29057820e210c2743f7" exitCode=0 Oct 30 19:12:16.139578 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:16.139055 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj" event={"ID":"a80bd19d-b0c9-48f8-a200-2c59d57c2ed1","Type":"ContainerDied","Data":"dcebb7b1348d6f7eaf2e71c2e341576b2186f15c07b2e29057820e210c2743f7"} Oct 30 19:12:17.314554 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:17.314525 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj" Oct 30 19:12:17.387691 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:17.387654 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a80bd19d-b0c9-48f8-a200-2c59d57c2ed1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a80bd19d-b0c9-48f8-a200-2c59d57c2ed1-maas-api-openshift-ca\") pod \"a80bd19d-b0c9-48f8-a200-2c59d57c2ed1\" (UID: \"a80bd19d-b0c9-48f8-a200-2c59d57c2ed1\") " Oct 30 19:12:17.387896 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:17.387786 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a80bd19d-b0c9-48f8-a200-2c59d57c2ed1-kube-api-access-5n4hq\" (UniqueName: \"kubernetes.io/projected/a80bd19d-b0c9-48f8-a200-2c59d57c2ed1-kube-api-access-5n4hq\") pod \"a80bd19d-b0c9-48f8-a200-2c59d57c2ed1\" (UID: \"a80bd19d-b0c9-48f8-a200-2c59d57c2ed1\") " Oct 30 19:12:17.387896 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:17.387849 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a80bd19d-b0c9-48f8-a200-2c59d57c2ed1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a80bd19d-b0c9-48f8-a200-2c59d57c2ed1-maas-api-xks-ca\") pod \"a80bd19d-b0c9-48f8-a200-2c59d57c2ed1\" (UID: \"a80bd19d-b0c9-48f8-a200-2c59d57c2ed1\") " Oct 30 19:12:17.388107 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:17.388084 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a80bd19d-b0c9-48f8-a200-2c59d57c2ed1-maas-api-openshift-ca" pod "a80bd19d-b0c9-48f8-a200-2c59d57c2ed1" (UID: "a80bd19d-b0c9-48f8-a200-2c59d57c2ed1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:17.388361 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:17.388343 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a80bd19d-b0c9-48f8-a200-2c59d57c2ed1-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:12:17.390473 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:17.390438 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a80bd19d-b0c9-48f8-a200-2c59d57c2ed1-kube-api-access-5n4hq" pod "a80bd19d-b0c9-48f8-a200-2c59d57c2ed1" (UID: "a80bd19d-b0c9-48f8-a200-2c59d57c2ed1"). InnerVolumeSpecName "kube-api-access-5n4hq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:17.390473 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:17.390448 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a80bd19d-b0c9-48f8-a200-2c59d57c2ed1-maas-api-xks-ca" pod "a80bd19d-b0c9-48f8-a200-2c59d57c2ed1" (UID: "a80bd19d-b0c9-48f8-a200-2c59d57c2ed1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:12:17.489663 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:17.489624 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n4hq\" (UniqueName: \"kubernetes.io/projected/a80bd19d-b0c9-48f8-a200-2c59d57c2ed1-kube-api-access-5n4hq\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:12:17.489663 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:17.489659 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a80bd19d-b0c9-48f8-a200-2c59d57c2ed1-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:12:18.152454 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:18.152408 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj" event={"ID":"a80bd19d-b0c9-48f8-a200-2c59d57c2ed1","Type":"ContainerDied","Data":"09db5db0d8e05875661278e0ffc40d3f1a9f4436a7735594055acdaf9acfd396"} Oct 30 19:12:18.152688 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:18.152458 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj" Oct 30 19:12:18.152688 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:18.152480 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09db5db0d8e05875661278e0ffc40d3f1a9f4436a7735594055acdaf9acfd396" Oct 30 19:12:26.403719 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:12:26.403680 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:12:51.448611 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:51.448528 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj"] Oct 30 19:12:51.498034 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:51.497987 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83ab2c-wj2jj"] Oct 30 19:12:52.099153 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:12:52.099102 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a80bd19d-b0c9-48f8-a200-2c59d57c2ed1" path="/var/lib/kubelet/pods/a80bd19d-b0c9-48f8-a200-2c59d57c2ed1/volumes" Oct 30 19:12:56.398975 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:12:56.398936 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:13:01.769236 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:01.769202 4531 scope.go:122] "RemoveContainer" containerID="c4cb4baf6e3e47ab912da1e2a511354dfc7f97229d9b7179a518224590ea3c32" Oct 30 19:13:04.678653 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:04.678617 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h"] Oct 30 19:13:04.679195 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:04.679170 4531 cpu_manager.go:414] "containerMap: removing container" podUID="a80bd19d-b0c9-48f8-a200-2c59d57c2ed1" containerName="revoke-keys" Oct 30 19:13:04.679195 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:04.679192 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a80bd19d-b0c9-48f8-a200-2c59d57c2ed1" containerName="revoke-keys" Oct 30 19:13:04.679385 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:04.679317 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a80bd19d-b0c9-48f8-a200-2c59d57c2ed1" containerName="revoke-keys" Oct 30 19:13:04.683874 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:04.683847 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h" Oct 30 19:13:04.738348 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:04.738305 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:13:04.787732 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:04.787695 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:13:04.787732 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:04.787724 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:13:04.787966 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:04.787794 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:13:04.790158 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:04.790122 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h"] Oct 30 19:13:04.853861 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:04.853829 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca527ce-f869-46c0-96d4-39bb22374810-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h\" (UID: \"5ca527ce-f869-46c0-96d4-39bb22374810\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h" Oct 30 19:13:04.854043 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:04.853888 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca527ce-f869-46c0-96d4-39bb22374810-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h\" (UID: \"5ca527ce-f869-46c0-96d4-39bb22374810\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h" Oct 30 19:13:04.854043 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:04.853973 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz54g\" (UniqueName: \"kubernetes.io/projected/5ca527ce-f869-46c0-96d4-39bb22374810-kube-api-access-wz54g\") pod \"maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h\" (UID: \"5ca527ce-f869-46c0-96d4-39bb22374810\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h" Oct 30 19:13:04.955156 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:04.955032 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca527ce-f869-46c0-96d4-39bb22374810-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h\" (UID: \"5ca527ce-f869-46c0-96d4-39bb22374810\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h" Oct 30 19:13:04.955355 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:04.955197 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca527ce-f869-46c0-96d4-39bb22374810-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h\" (UID: \"5ca527ce-f869-46c0-96d4-39bb22374810\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h" Oct 30 19:13:04.955355 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:04.955268 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz54g\" (UniqueName: \"kubernetes.io/projected/5ca527ce-f869-46c0-96d4-39bb22374810-kube-api-access-wz54g\") pod \"maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h\" (UID: \"5ca527ce-f869-46c0-96d4-39bb22374810\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h" Oct 30 19:13:04.955959 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:04.955926 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca527ce-f869-46c0-96d4-39bb22374810-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h\" (UID: \"5ca527ce-f869-46c0-96d4-39bb22374810\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h" Oct 30 19:13:04.958192 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:04.958162 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca527ce-f869-46c0-96d4-39bb22374810-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h\" (UID: \"5ca527ce-f869-46c0-96d4-39bb22374810\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h" Oct 30 19:13:05.018088 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:05.018047 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz54g\" (UniqueName: \"kubernetes.io/projected/5ca527ce-f869-46c0-96d4-39bb22374810-kube-api-access-wz54g\") pod \"maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h\" (UID: \"5ca527ce-f869-46c0-96d4-39bb22374810\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h" Oct 30 19:13:05.297241 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:05.297202 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h" Oct 30 19:13:05.681770 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:05.681735 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h"] Oct 30 19:13:05.682577 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:13:05.682546 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca527ce_f869_46c0_96d4_39bb22374810.slice/crio-0843df637f610bfca41d436352d8604af2501102e86a187744a31f58dbd24738 WatchSource:0}: Error finding container 0843df637f610bfca41d436352d8604af2501102e86a187744a31f58dbd24738: Status 404 returned error can't find the container with id 0843df637f610bfca41d436352d8604af2501102e86a187744a31f58dbd24738 Oct 30 19:13:06.438860 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:06.438775 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h" event={"ID":"5ca527ce-f869-46c0-96d4-39bb22374810","Type":"ContainerStarted","Data":"1f40e9b529530d7e7268a6576b4b7b2718a9e9e461e54b2b86496a9007bef38d"} Oct 30 19:13:06.438860 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:06.438830 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h" event={"ID":"5ca527ce-f869-46c0-96d4-39bb22374810","Type":"ContainerStarted","Data":"0843df637f610bfca41d436352d8604af2501102e86a187744a31f58dbd24738"} Oct 30 19:13:06.559401 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:06.559332 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h" podStartSLOduration=2.55930821 podStartE2EDuration="2.55930821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:06.557707695 +0000 UTC m=+2346.584812655" watchObservedRunningTime="2026-10-30 19:13:06.55930821 +0000 UTC m=+2346.586413168" Oct 30 19:13:07.446827 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:07.446792 4531 generic.go:356] "Generic (PLEG): container finished" podID="5ca527ce-f869-46c0-96d4-39bb22374810" containerID="1f40e9b529530d7e7268a6576b4b7b2718a9e9e461e54b2b86496a9007bef38d" exitCode=0 Oct 30 19:13:07.447307 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:07.446883 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h" event={"ID":"5ca527ce-f869-46c0-96d4-39bb22374810","Type":"ContainerDied","Data":"1f40e9b529530d7e7268a6576b4b7b2718a9e9e461e54b2b86496a9007bef38d"} Oct 30 19:13:08.614940 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:08.614912 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h" Oct 30 19:13:08.711069 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:08.710977 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ca527ce-f869-46c0-96d4-39bb22374810-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca527ce-f869-46c0-96d4-39bb22374810-maas-api-openshift-ca\") pod \"5ca527ce-f869-46c0-96d4-39bb22374810\" (UID: \"5ca527ce-f869-46c0-96d4-39bb22374810\") " Oct 30 19:13:08.711270 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:08.711068 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ca527ce-f869-46c0-96d4-39bb22374810-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca527ce-f869-46c0-96d4-39bb22374810-maas-api-xks-ca\") pod \"5ca527ce-f869-46c0-96d4-39bb22374810\" (UID: \"5ca527ce-f869-46c0-96d4-39bb22374810\") " Oct 30 19:13:08.711270 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:08.711189 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ca527ce-f869-46c0-96d4-39bb22374810-kube-api-access-wz54g\" (UniqueName: \"kubernetes.io/projected/5ca527ce-f869-46c0-96d4-39bb22374810-kube-api-access-wz54g\") pod \"5ca527ce-f869-46c0-96d4-39bb22374810\" (UID: \"5ca527ce-f869-46c0-96d4-39bb22374810\") " Oct 30 19:13:08.711441 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:08.711415 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ca527ce-f869-46c0-96d4-39bb22374810-maas-api-openshift-ca" pod "5ca527ce-f869-46c0-96d4-39bb22374810" (UID: "5ca527ce-f869-46c0-96d4-39bb22374810"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:08.713522 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:08.713495 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca527ce-f869-46c0-96d4-39bb22374810-kube-api-access-wz54g" pod "5ca527ce-f869-46c0-96d4-39bb22374810" (UID: "5ca527ce-f869-46c0-96d4-39bb22374810"). InnerVolumeSpecName "kube-api-access-wz54g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:08.713636 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:08.713496 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ca527ce-f869-46c0-96d4-39bb22374810-maas-api-xks-ca" pod "5ca527ce-f869-46c0-96d4-39bb22374810" (UID: "5ca527ce-f869-46c0-96d4-39bb22374810"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:13:08.812794 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:08.812746 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz54g\" (UniqueName: \"kubernetes.io/projected/5ca527ce-f869-46c0-96d4-39bb22374810-kube-api-access-wz54g\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:13:08.812794 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:08.812781 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca527ce-f869-46c0-96d4-39bb22374810-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:13:08.812794 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:08.812794 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca527ce-f869-46c0-96d4-39bb22374810-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:13:09.460112 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:09.460083 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h" Oct 30 19:13:09.460321 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:09.460076 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h" event={"ID":"5ca527ce-f869-46c0-96d4-39bb22374810","Type":"ContainerDied","Data":"0843df637f610bfca41d436352d8604af2501102e86a187744a31f58dbd24738"} Oct 30 19:13:09.460321 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:09.460176 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0843df637f610bfca41d436352d8604af2501102e86a187744a31f58dbd24738" Oct 30 19:13:16.899381 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:16.899336 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms"] Oct 30 19:13:21.900312 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:21.900259 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms" podUID="7337eb2d-f4b3-44a4-a2bf-3d7557522984" containerName="payload-processing" containerID="cri-o://c9c7226ad12dbfda46788e7aa0e58ae0dcea5446eceef6dc2cda2377b9740009" gracePeriod=25 Oct 30 19:13:22.202916 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:22.202886 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms" Oct 30 19:13:22.292320 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:22.292270 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7337eb2d-f4b3-44a4-a2bf-3d7557522984-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7337eb2d-f4b3-44a4-a2bf-3d7557522984-plugins-config-volume\") pod \"7337eb2d-f4b3-44a4-a2bf-3d7557522984\" (UID: \"7337eb2d-f4b3-44a4-a2bf-3d7557522984\") " Oct 30 19:13:22.292514 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:22.292453 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7337eb2d-f4b3-44a4-a2bf-3d7557522984-kube-api-access-hb86n\" (UniqueName: \"kubernetes.io/projected/7337eb2d-f4b3-44a4-a2bf-3d7557522984-kube-api-access-hb86n\") pod \"7337eb2d-f4b3-44a4-a2bf-3d7557522984\" (UID: \"7337eb2d-f4b3-44a4-a2bf-3d7557522984\") " Oct 30 19:13:22.292740 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:22.292716 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7337eb2d-f4b3-44a4-a2bf-3d7557522984-plugins-config-volume" pod "7337eb2d-f4b3-44a4-a2bf-3d7557522984" (UID: "7337eb2d-f4b3-44a4-a2bf-3d7557522984"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:22.292999 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:22.292974 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7337eb2d-f4b3-44a4-a2bf-3d7557522984-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:13:22.294949 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:22.294912 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7337eb2d-f4b3-44a4-a2bf-3d7557522984-kube-api-access-hb86n" pod "7337eb2d-f4b3-44a4-a2bf-3d7557522984" (UID: "7337eb2d-f4b3-44a4-a2bf-3d7557522984"). InnerVolumeSpecName "kube-api-access-hb86n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:22.394593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:22.394544 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb86n\" (UniqueName: \"kubernetes.io/projected/7337eb2d-f4b3-44a4-a2bf-3d7557522984-kube-api-access-hb86n\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:13:22.541884 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:22.541846 4531 generic.go:356] "Generic (PLEG): container finished" podID="7337eb2d-f4b3-44a4-a2bf-3d7557522984" containerID="c9c7226ad12dbfda46788e7aa0e58ae0dcea5446eceef6dc2cda2377b9740009" exitCode=0 Oct 30 19:13:22.542094 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:22.541905 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms" Oct 30 19:13:22.542094 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:22.541923 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms" event={"ID":"7337eb2d-f4b3-44a4-a2bf-3d7557522984","Type":"ContainerDied","Data":"c9c7226ad12dbfda46788e7aa0e58ae0dcea5446eceef6dc2cda2377b9740009"} Oct 30 19:13:22.542094 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:22.541979 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms" event={"ID":"7337eb2d-f4b3-44a4-a2bf-3d7557522984","Type":"ContainerDied","Data":"e58f1cd8fcf9cbf8da832369824a07ac8d92a6b63218544f69b127a421a8fa6e"} Oct 30 19:13:22.542094 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:22.542008 4531 scope.go:122] "RemoveContainer" containerID="c9c7226ad12dbfda46788e7aa0e58ae0dcea5446eceef6dc2cda2377b9740009" Oct 30 19:13:22.552701 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:22.552673 4531 scope.go:122] "RemoveContainer" containerID="c9c7226ad12dbfda46788e7aa0e58ae0dcea5446eceef6dc2cda2377b9740009" Oct 30 19:13:22.553046 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:13:22.553021 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9c7226ad12dbfda46788e7aa0e58ae0dcea5446eceef6dc2cda2377b9740009\": container with ID starting with c9c7226ad12dbfda46788e7aa0e58ae0dcea5446eceef6dc2cda2377b9740009 not found: ID does not exist" containerID="c9c7226ad12dbfda46788e7aa0e58ae0dcea5446eceef6dc2cda2377b9740009" Oct 30 19:13:22.553116 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:22.553055 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9c7226ad12dbfda46788e7aa0e58ae0dcea5446eceef6dc2cda2377b9740009"} err="failed to get container status \"c9c7226ad12dbfda46788e7aa0e58ae0dcea5446eceef6dc2cda2377b9740009\": rpc error: code = NotFound desc = could not find container \"c9c7226ad12dbfda46788e7aa0e58ae0dcea5446eceef6dc2cda2377b9740009\": container with ID starting with c9c7226ad12dbfda46788e7aa0e58ae0dcea5446eceef6dc2cda2377b9740009 not found: ID does not exist" Oct 30 19:13:22.709632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:22.709588 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms"] Oct 30 19:13:22.759167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:22.759090 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5a035d-555dddc986-gqzms"] Oct 30 19:13:23.750424 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:23.750381 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49"] Oct 30 19:13:23.750903 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:23.750611 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" podUID="f9341cd2-23c8-4c4e-a895-2529c03dad3c" containerName="istio-proxy" containerID="cri-o://16bc92296e0f13c8d46eeafbaede39fda7bbbe57c587621ea1b559bb901bf525" gracePeriod=30 Oct 30 19:13:24.098912 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:24.098826 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7337eb2d-f4b3-44a4-a2bf-3d7557522984" path="/var/lib/kubelet/pods/7337eb2d-f4b3-44a4-a2bf-3d7557522984/volumes" Oct 30 19:13:24.996427 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:24.996388 4531 patch_prober.go:29] interesting pod/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:13:24.996874 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:24.996455 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" podUID="f9341cd2-23c8-4c4e-a895-2529c03dad3c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:13:26.399191 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:13:26.399150 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:13:29.023776 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.023702 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 19:13:29.072285 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.072247 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9341cd2-23c8-4c4e-a895-2529c03dad3c-kube-api-access-vn9mh\" (UniqueName: \"kubernetes.io/projected/f9341cd2-23c8-4c4e-a895-2529c03dad3c-kube-api-access-vn9mh\") pod \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " Oct 30 19:13:29.072459 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.072297 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-workload-socket\") pod \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " Oct 30 19:13:29.072459 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.072361 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-credential-socket\") pod \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " Oct 30 19:13:29.072459 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.072411 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-envoy\") pod \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " Oct 30 19:13:29.072569 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.072490 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-token\" (UniqueName: \"kubernetes.io/projected/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-token\") pod \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " Oct 30 19:13:29.072569 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.072543 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-data\") pod \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " Oct 30 19:13:29.072664 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.072596 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-podinfo\") pod \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " Oct 30 19:13:29.072664 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.072634 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-workload-certs\") pod \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " Oct 30 19:13:29.072664 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.072630 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-workload-socket" pod "f9341cd2-23c8-4c4e-a895-2529c03dad3c" (UID: "f9341cd2-23c8-4c4e-a895-2529c03dad3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:29.072827 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.072689 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istiod-ca-cert\") pod \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\" (UID: \"f9341cd2-23c8-4c4e-a895-2529c03dad3c\") " Oct 30 19:13:29.072827 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.072687 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-credential-socket" pod "f9341cd2-23c8-4c4e-a895-2529c03dad3c" (UID: "f9341cd2-23c8-4c4e-a895-2529c03dad3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:29.072940 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.072902 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-data" pod "f9341cd2-23c8-4c4e-a895-2529c03dad3c" (UID: "f9341cd2-23c8-4c4e-a895-2529c03dad3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:29.072940 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.072920 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-workload-certs" pod "f9341cd2-23c8-4c4e-a895-2529c03dad3c" (UID: "f9341cd2-23c8-4c4e-a895-2529c03dad3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:29.073488 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.073308 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:13:29.073488 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.073334 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:13:29.073488 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.073350 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:13:29.073488 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.073364 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:13:29.073488 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.073397 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istiod-ca-cert" pod "f9341cd2-23c8-4c4e-a895-2529c03dad3c" (UID: "f9341cd2-23c8-4c4e-a895-2529c03dad3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:29.074814 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.074785 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-envoy" pod "f9341cd2-23c8-4c4e-a895-2529c03dad3c" (UID: "f9341cd2-23c8-4c4e-a895-2529c03dad3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:29.074908 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.074807 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9341cd2-23c8-4c4e-a895-2529c03dad3c-kube-api-access-vn9mh" pod "f9341cd2-23c8-4c4e-a895-2529c03dad3c" (UID: "f9341cd2-23c8-4c4e-a895-2529c03dad3c"). InnerVolumeSpecName "kube-api-access-vn9mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:29.074908 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.074859 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-token" pod "f9341cd2-23c8-4c4e-a895-2529c03dad3c" (UID: "f9341cd2-23c8-4c4e-a895-2529c03dad3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:29.074996 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.074962 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-podinfo" pod "f9341cd2-23c8-4c4e-a895-2529c03dad3c" (UID: "f9341cd2-23c8-4c4e-a895-2529c03dad3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:13:29.173952 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.173914 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:13:29.173952 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.173944 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:13:29.173952 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.173955 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn9mh\" (UniqueName: \"kubernetes.io/projected/f9341cd2-23c8-4c4e-a895-2529c03dad3c-kube-api-access-vn9mh\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:13:29.174217 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.173965 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:13:29.174217 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.173975 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9341cd2-23c8-4c4e-a895-2529c03dad3c-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:13:29.585705 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.585668 4531 generic.go:356] "Generic (PLEG): container finished" podID="f9341cd2-23c8-4c4e-a895-2529c03dad3c" containerID="16bc92296e0f13c8d46eeafbaede39fda7bbbe57c587621ea1b559bb901bf525" exitCode=0 Oct 30 19:13:29.585884 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.585712 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" event={"ID":"f9341cd2-23c8-4c4e-a895-2529c03dad3c","Type":"ContainerDied","Data":"16bc92296e0f13c8d46eeafbaede39fda7bbbe57c587621ea1b559bb901bf525"} Oct 30 19:13:29.585884 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.585757 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" event={"ID":"f9341cd2-23c8-4c4e-a895-2529c03dad3c","Type":"ContainerDied","Data":"0ec8330f7cd2067c970b12bfc3765a841e9efdb44870fe4c11a5a3e54bfc31f4"} Oct 30 19:13:29.585884 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.585779 4531 scope.go:122] "RemoveContainer" containerID="16bc92296e0f13c8d46eeafbaede39fda7bbbe57c587621ea1b559bb901bf525" Oct 30 19:13:29.585884 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.585713 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49" Oct 30 19:13:29.596233 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.596204 4531 scope.go:122] "RemoveContainer" containerID="16bc92296e0f13c8d46eeafbaede39fda7bbbe57c587621ea1b559bb901bf525" Oct 30 19:13:29.596530 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:13:29.596508 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16bc92296e0f13c8d46eeafbaede39fda7bbbe57c587621ea1b559bb901bf525\": container with ID starting with 16bc92296e0f13c8d46eeafbaede39fda7bbbe57c587621ea1b559bb901bf525 not found: ID does not exist" containerID="16bc92296e0f13c8d46eeafbaede39fda7bbbe57c587621ea1b559bb901bf525" Oct 30 19:13:29.596589 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.596539 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16bc92296e0f13c8d46eeafbaede39fda7bbbe57c587621ea1b559bb901bf525"} err="failed to get container status \"16bc92296e0f13c8d46eeafbaede39fda7bbbe57c587621ea1b559bb901bf525\": rpc error: code = NotFound desc = could not find container \"16bc92296e0f13c8d46eeafbaede39fda7bbbe57c587621ea1b559bb901bf525\": container with ID starting with 16bc92296e0f13c8d46eeafbaede39fda7bbbe57c587621ea1b559bb901bf525 not found: ID does not exist" Oct 30 19:13:29.752769 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.752735 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49"] Oct 30 19:13:29.759224 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:29.759197 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-lxs49"] Oct 30 19:13:30.098931 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:30.098897 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9341cd2-23c8-4c4e-a895-2529c03dad3c" path="/var/lib/kubelet/pods/f9341cd2-23c8-4c4e-a895-2529c03dad3c/volumes" Oct 30 19:13:37.243051 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:37.243006 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv"] Oct 30 19:13:37.243577 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:37.243272 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" podUID="ef9e8996-dad3-4888-821f-2ec713398074" containerName="istio-proxy" containerID="cri-o://1de00f85dd2ecd987277d0c1d0b3139fd3116cdb99671f82353d644582c57f1d" gracePeriod=30 Oct 30 19:13:37.615155 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:37.615054 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:13:37.615306 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:37.615145 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" podUID="ef9e8996-dad3-4888-821f-2ec713398074" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:13:39.004198 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:39.004166 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs_a9ace4a6-85c4-477a-99d5-0995681b0bb7/payload-processing/0.log" Oct 30 19:13:39.819068 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:39.819035 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62_5bbecb8e-40ce-4b0d-948f-c7b53598da3f/payload-processing/0.log" Oct 30 19:13:40.048576 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:40.048545 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62_5bbecb8e-40ce-4b0d-948f-c7b53598da3f/payload-processing/0.log" Oct 30 19:13:41.742791 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:41.742697 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd"] Oct 30 19:13:41.743246 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:41.743201 4531 cpu_manager.go:414] "containerMap: removing container" podUID="f9341cd2-23c8-4c4e-a895-2529c03dad3c" containerName="istio-proxy" Oct 30 19:13:41.743246 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:41.743219 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9341cd2-23c8-4c4e-a895-2529c03dad3c" containerName="istio-proxy" Oct 30 19:13:41.743246 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:41.743244 4531 cpu_manager.go:414] "containerMap: removing container" podUID="7337eb2d-f4b3-44a4-a2bf-3d7557522984" containerName="payload-processing" Oct 30 19:13:41.743373 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:41.743250 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7337eb2d-f4b3-44a4-a2bf-3d7557522984" containerName="payload-processing" Oct 30 19:13:41.743373 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:41.743263 4531 cpu_manager.go:414] "containerMap: removing container" podUID="5ca527ce-f869-46c0-96d4-39bb22374810" containerName="revoke-keys" Oct 30 19:13:41.743373 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:41.743267 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ca527ce-f869-46c0-96d4-39bb22374810" containerName="revoke-keys" Oct 30 19:13:41.743373 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:41.743359 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9341cd2-23c8-4c4e-a895-2529c03dad3c" containerName="istio-proxy" Oct 30 19:13:41.743373 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:41.743369 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7337eb2d-f4b3-44a4-a2bf-3d7557522984" containerName="payload-processing" Oct 30 19:13:41.743562 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:41.743376 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ca527ce-f869-46c0-96d4-39bb22374810" containerName="revoke-keys" Oct 30 19:13:41.748265 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:41.748238 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd" Oct 30 19:13:41.832914 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:41.832871 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef6f5e4-3502-42f1-a565-a19a131e2a2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd\" (UID: \"6ef6f5e4-3502-42f1-a565-a19a131e2a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd" Oct 30 19:13:41.833098 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:41.833067 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef6f5e4-3502-42f1-a565-a19a131e2a2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd\" (UID: \"6ef6f5e4-3502-42f1-a565-a19a131e2a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd" Oct 30 19:13:41.833232 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:41.833212 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl8m4\" (UniqueName: \"kubernetes.io/projected/6ef6f5e4-3502-42f1-a565-a19a131e2a2a-kube-api-access-wl8m4\") pod \"maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd\" (UID: \"6ef6f5e4-3502-42f1-a565-a19a131e2a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd" Oct 30 19:13:41.852264 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:41.852223 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:13:41.852264 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:41.852253 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:13:41.852264 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:41.852269 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:13:41.852529 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:41.852315 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:13:41.854882 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:41.854853 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd"] Oct 30 19:13:41.934939 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:41.934900 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl8m4\" (UniqueName: \"kubernetes.io/projected/6ef6f5e4-3502-42f1-a565-a19a131e2a2a-kube-api-access-wl8m4\") pod \"maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd\" (UID: \"6ef6f5e4-3502-42f1-a565-a19a131e2a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd" Oct 30 19:13:41.935201 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:41.934958 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef6f5e4-3502-42f1-a565-a19a131e2a2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd\" (UID: \"6ef6f5e4-3502-42f1-a565-a19a131e2a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd" Oct 30 19:13:41.935201 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:41.935035 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef6f5e4-3502-42f1-a565-a19a131e2a2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd\" (UID: \"6ef6f5e4-3502-42f1-a565-a19a131e2a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd" Oct 30 19:13:41.935707 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:41.935682 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef6f5e4-3502-42f1-a565-a19a131e2a2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd\" (UID: \"6ef6f5e4-3502-42f1-a565-a19a131e2a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd" Oct 30 19:13:41.937621 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:41.937597 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef6f5e4-3502-42f1-a565-a19a131e2a2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd\" (UID: \"6ef6f5e4-3502-42f1-a565-a19a131e2a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd" Oct 30 19:13:41.999302 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:41.999219 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl8m4\" (UniqueName: \"kubernetes.io/projected/6ef6f5e4-3502-42f1-a565-a19a131e2a2a-kube-api-access-wl8m4\") pod \"maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd\" (UID: \"6ef6f5e4-3502-42f1-a565-a19a131e2a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd" Oct 30 19:13:42.062536 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.062500 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd" Oct 30 19:13:42.448875 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:13:42.448841 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef6f5e4_3502_42f1_a565_a19a131e2a2a.slice/crio-7b75a8d5feeade10ccc38191b55f299f038530da43f28a2c27b64f4777fa9e59 WatchSource:0}: Error finding container 7b75a8d5feeade10ccc38191b55f299f038530da43f28a2c27b64f4777fa9e59: Status 404 returned error can't find the container with id 7b75a8d5feeade10ccc38191b55f299f038530da43f28a2c27b64f4777fa9e59 Oct 30 19:13:42.449074 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.448965 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd"] Oct 30 19:13:42.502677 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.502645 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 19:13:42.542077 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.541974 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-workload-certs\") pod \"ef9e8996-dad3-4888-821f-2ec713398074\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " Oct 30 19:13:42.542077 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.542023 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef9e8996-dad3-4888-821f-2ec713398074-istio-token\" (UniqueName: \"kubernetes.io/projected/ef9e8996-dad3-4888-821f-2ec713398074-istio-token\") pod \"ef9e8996-dad3-4888-821f-2ec713398074\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " Oct 30 19:13:42.542077 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.542056 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-istio-data\") pod \"ef9e8996-dad3-4888-821f-2ec713398074\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " Oct 30 19:13:42.542077 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.542074 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-istio-envoy\") pod \"ef9e8996-dad3-4888-821f-2ec713398074\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " Oct 30 19:13:42.542470 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.542166 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef9e8996-dad3-4888-821f-2ec713398074-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9e8996-dad3-4888-821f-2ec713398074-istio-podinfo\") pod \"ef9e8996-dad3-4888-821f-2ec713398074\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " Oct 30 19:13:42.542470 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.542238 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-credential-socket\") pod \"ef9e8996-dad3-4888-821f-2ec713398074\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " Oct 30 19:13:42.542470 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.542296 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef9e8996-dad3-4888-821f-2ec713398074-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9e8996-dad3-4888-821f-2ec713398074-istiod-ca-cert\") pod \"ef9e8996-dad3-4888-821f-2ec713398074\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " Oct 30 19:13:42.542470 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.542351 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-workload-certs" pod "ef9e8996-dad3-4888-821f-2ec713398074" (UID: "ef9e8996-dad3-4888-821f-2ec713398074"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:42.542470 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.542381 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef9e8996-dad3-4888-821f-2ec713398074-kube-api-access-7p6qv\" (UniqueName: \"kubernetes.io/projected/ef9e8996-dad3-4888-821f-2ec713398074-kube-api-access-7p6qv\") pod \"ef9e8996-dad3-4888-821f-2ec713398074\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " Oct 30 19:13:42.542470 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.542452 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-istio-data" pod "ef9e8996-dad3-4888-821f-2ec713398074" (UID: "ef9e8996-dad3-4888-821f-2ec713398074"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:42.542777 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.542502 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-workload-socket\") pod \"ef9e8996-dad3-4888-821f-2ec713398074\" (UID: \"ef9e8996-dad3-4888-821f-2ec713398074\") " Oct 30 19:13:42.542777 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.542659 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-credential-socket" pod "ef9e8996-dad3-4888-821f-2ec713398074" (UID: "ef9e8996-dad3-4888-821f-2ec713398074"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:42.542893 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.542853 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-workload-socket" pod "ef9e8996-dad3-4888-821f-2ec713398074" (UID: "ef9e8996-dad3-4888-821f-2ec713398074"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:42.543123 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.543098 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9e8996-dad3-4888-821f-2ec713398074-istiod-ca-cert" pod "ef9e8996-dad3-4888-821f-2ec713398074" (UID: "ef9e8996-dad3-4888-821f-2ec713398074"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:42.543273 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.543251 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:13:42.543335 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.543282 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef9e8996-dad3-4888-821f-2ec713398074-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:13:42.543335 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.543296 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:13:42.543335 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.543310 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:13:42.543335 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.543325 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:13:42.544798 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.544771 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-istio-envoy" pod "ef9e8996-dad3-4888-821f-2ec713398074" (UID: "ef9e8996-dad3-4888-821f-2ec713398074"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:42.544798 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.544791 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9e8996-dad3-4888-821f-2ec713398074-kube-api-access-7p6qv" pod "ef9e8996-dad3-4888-821f-2ec713398074" (UID: "ef9e8996-dad3-4888-821f-2ec713398074"). InnerVolumeSpecName "kube-api-access-7p6qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:42.544977 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.544827 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9e8996-dad3-4888-821f-2ec713398074-istio-token" pod "ef9e8996-dad3-4888-821f-2ec713398074" (UID: "ef9e8996-dad3-4888-821f-2ec713398074"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:42.545143 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.545108 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef9e8996-dad3-4888-821f-2ec713398074-istio-podinfo" pod "ef9e8996-dad3-4888-821f-2ec713398074" (UID: "ef9e8996-dad3-4888-821f-2ec713398074"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:13:42.644837 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.644794 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef9e8996-dad3-4888-821f-2ec713398074-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:13:42.644837 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.644839 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p6qv\" (UniqueName: \"kubernetes.io/projected/ef9e8996-dad3-4888-821f-2ec713398074-kube-api-access-7p6qv\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:13:42.645159 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.644855 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef9e8996-dad3-4888-821f-2ec713398074-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:13:42.645159 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.644870 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef9e8996-dad3-4888-821f-2ec713398074-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:13:42.678493 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.678455 4531 generic.go:356] "Generic (PLEG): container finished" podID="ef9e8996-dad3-4888-821f-2ec713398074" containerID="1de00f85dd2ecd987277d0c1d0b3139fd3116cdb99671f82353d644582c57f1d" exitCode=0 Oct 30 19:13:42.678684 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.678512 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" Oct 30 19:13:42.678684 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.678530 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" event={"ID":"ef9e8996-dad3-4888-821f-2ec713398074","Type":"ContainerDied","Data":"1de00f85dd2ecd987277d0c1d0b3139fd3116cdb99671f82353d644582c57f1d"} Oct 30 19:13:42.678684 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.678593 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv" event={"ID":"ef9e8996-dad3-4888-821f-2ec713398074","Type":"ContainerDied","Data":"7f5ef83295b6eff5e72706064e993575402626651bdde96100790ce408123012"} Oct 30 19:13:42.678684 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.678626 4531 scope.go:122] "RemoveContainer" containerID="1de00f85dd2ecd987277d0c1d0b3139fd3116cdb99671f82353d644582c57f1d" Oct 30 19:13:42.680193 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.680163 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd" event={"ID":"6ef6f5e4-3502-42f1-a565-a19a131e2a2a","Type":"ContainerStarted","Data":"7b75a8d5feeade10ccc38191b55f299f038530da43f28a2c27b64f4777fa9e59"} Oct 30 19:13:42.689414 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.689389 4531 scope.go:122] "RemoveContainer" containerID="1de00f85dd2ecd987277d0c1d0b3139fd3116cdb99671f82353d644582c57f1d" Oct 30 19:13:42.689777 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:13:42.689752 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1de00f85dd2ecd987277d0c1d0b3139fd3116cdb99671f82353d644582c57f1d\": container with ID starting with 1de00f85dd2ecd987277d0c1d0b3139fd3116cdb99671f82353d644582c57f1d not found: ID does not exist" containerID="1de00f85dd2ecd987277d0c1d0b3139fd3116cdb99671f82353d644582c57f1d" Oct 30 19:13:42.689843 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.689788 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de00f85dd2ecd987277d0c1d0b3139fd3116cdb99671f82353d644582c57f1d"} err="failed to get container status \"1de00f85dd2ecd987277d0c1d0b3139fd3116cdb99671f82353d644582c57f1d\": rpc error: code = NotFound desc = could not find container \"1de00f85dd2ecd987277d0c1d0b3139fd3116cdb99671f82353d644582c57f1d\": container with ID starting with 1de00f85dd2ecd987277d0c1d0b3139fd3116cdb99671f82353d644582c57f1d not found: ID does not exist" Oct 30 19:13:42.849340 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.849242 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv"] Oct 30 19:13:42.851706 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:42.851673 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-l55pv"] Oct 30 19:13:43.689336 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:43.689245 4531 generic.go:356] "Generic (PLEG): container finished" podID="6ef6f5e4-3502-42f1-a565-a19a131e2a2a" containerID="fba1f919b6513d3568d5243bcd458e5ed22b6d65d8798d0f35a1c6f34fbf194f" exitCode=0 Oct 30 19:13:43.689486 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:43.689342 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd" event={"ID":"6ef6f5e4-3502-42f1-a565-a19a131e2a2a","Type":"ContainerDied","Data":"fba1f919b6513d3568d5243bcd458e5ed22b6d65d8798d0f35a1c6f34fbf194f"} Oct 30 19:13:44.099236 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:44.099200 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef9e8996-dad3-4888-821f-2ec713398074" path="/var/lib/kubelet/pods/ef9e8996-dad3-4888-821f-2ec713398074/volumes" Oct 30 19:13:44.837426 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:44.837390 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd" Oct 30 19:13:44.971443 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:44.971341 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ef6f5e4-3502-42f1-a565-a19a131e2a2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef6f5e4-3502-42f1-a565-a19a131e2a2a-maas-api-openshift-ca\") pod \"6ef6f5e4-3502-42f1-a565-a19a131e2a2a\" (UID: \"6ef6f5e4-3502-42f1-a565-a19a131e2a2a\") " Oct 30 19:13:44.971443 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:44.971384 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef6f5e4-3502-42f1-a565-a19a131e2a2a-kube-api-access-wl8m4\" (UniqueName: \"kubernetes.io/projected/6ef6f5e4-3502-42f1-a565-a19a131e2a2a-kube-api-access-wl8m4\") pod \"6ef6f5e4-3502-42f1-a565-a19a131e2a2a\" (UID: \"6ef6f5e4-3502-42f1-a565-a19a131e2a2a\") " Oct 30 19:13:44.971695 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:44.971578 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ef6f5e4-3502-42f1-a565-a19a131e2a2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef6f5e4-3502-42f1-a565-a19a131e2a2a-maas-api-xks-ca\") pod \"6ef6f5e4-3502-42f1-a565-a19a131e2a2a\" (UID: \"6ef6f5e4-3502-42f1-a565-a19a131e2a2a\") " Oct 30 19:13:44.971763 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:44.971723 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef6f5e4-3502-42f1-a565-a19a131e2a2a-maas-api-openshift-ca" pod "6ef6f5e4-3502-42f1-a565-a19a131e2a2a" (UID: "6ef6f5e4-3502-42f1-a565-a19a131e2a2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:44.971973 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:44.971949 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef6f5e4-3502-42f1-a565-a19a131e2a2a-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:13:44.973873 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:44.973845 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ef6f5e4-3502-42f1-a565-a19a131e2a2a-maas-api-xks-ca" pod "6ef6f5e4-3502-42f1-a565-a19a131e2a2a" (UID: "6ef6f5e4-3502-42f1-a565-a19a131e2a2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:13:44.973971 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:44.973886 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef6f5e4-3502-42f1-a565-a19a131e2a2a-kube-api-access-wl8m4" pod "6ef6f5e4-3502-42f1-a565-a19a131e2a2a" (UID: "6ef6f5e4-3502-42f1-a565-a19a131e2a2a"). InnerVolumeSpecName "kube-api-access-wl8m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:45.073077 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:45.073032 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef6f5e4-3502-42f1-a565-a19a131e2a2a-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:13:45.073077 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:45.073071 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl8m4\" (UniqueName: \"kubernetes.io/projected/6ef6f5e4-3502-42f1-a565-a19a131e2a2a-kube-api-access-wl8m4\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:13:45.703289 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:45.703243 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd" event={"ID":"6ef6f5e4-3502-42f1-a565-a19a131e2a2a","Type":"ContainerDied","Data":"7b75a8d5feeade10ccc38191b55f299f038530da43f28a2c27b64f4777fa9e59"} Oct 30 19:13:45.703289 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:45.703269 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd" Oct 30 19:13:45.703921 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:13:45.703312 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b75a8d5feeade10ccc38191b55f299f038530da43f28a2c27b64f4777fa9e59" Oct 30 19:13:56.401661 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:13:56.401618 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:14:01.792352 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:01.792308 4531 scope.go:122] "RemoveContainer" containerID="1ee2d85a72dcba966e88dff5e31445096b49a7445ab163ca4c9a04683336e616" Oct 30 19:14:01.802108 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:01.802084 4531 scope.go:122] "RemoveContainer" containerID="d0f8adc933bca5440734098d4d4a085dfd60916753b03a12dbe6b9898280f455" Oct 30 19:14:01.811428 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:01.811401 4531 scope.go:122] "RemoveContainer" containerID="1e58b95edf25fcc3734abbd560994f57e30dcbc5c494a6a8727c8bbecd2efc08" Oct 30 19:14:21.741757 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:21.741717 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h"] Oct 30 19:14:21.790898 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:21.790857 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5a035d-bfs9h"] Oct 30 19:14:22.098900 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:22.098864 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ca527ce-f869-46c0-96d4-39bb22374810" path="/var/lib/kubelet/pods/5ca527ce-f869-46c0-96d4-39bb22374810/volumes" Oct 30 19:14:26.400202 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:14:26.400158 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:14:29.341383 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:29.341335 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c"] Oct 30 19:14:29.341952 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:29.341646 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" podUID="cac0c9e8-98a5-49b1-93c4-944bfda920a2" containerName="istio-proxy" containerID="cri-o://c6f6e39e85a67bdf500076d74258c06679610567828a33329ce5a2f251ca9030" gracePeriod=30 Oct 30 19:14:29.892390 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:29.892355 4531 patch_prober.go:29] interesting pod/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:14:29.892567 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:29.892423 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" podUID="cac0c9e8-98a5-49b1-93c4-944bfda920a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:14:34.422889 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.422855 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7"] Oct 30 19:14:34.423304 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.423285 4531 cpu_manager.go:414] "containerMap: removing container" podUID="ef9e8996-dad3-4888-821f-2ec713398074" containerName="istio-proxy" Oct 30 19:14:34.423304 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.423296 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef9e8996-dad3-4888-821f-2ec713398074" containerName="istio-proxy" Oct 30 19:14:34.423410 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.423329 4531 cpu_manager.go:414] "containerMap: removing container" podUID="6ef6f5e4-3502-42f1-a565-a19a131e2a2a" containerName="revoke-keys" Oct 30 19:14:34.423410 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.423334 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef6f5e4-3502-42f1-a565-a19a131e2a2a" containerName="revoke-keys" Oct 30 19:14:34.423410 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.423403 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef6f5e4-3502-42f1-a565-a19a131e2a2a" containerName="revoke-keys" Oct 30 19:14:34.423522 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.423415 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef9e8996-dad3-4888-821f-2ec713398074" containerName="istio-proxy" Oct 30 19:14:34.426560 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.426544 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7" Oct 30 19:14:34.524327 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.524283 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af10d82c-0a5c-4139-958c-47721326cf4c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7\" (UID: \"af10d82c-0a5c-4139-958c-47721326cf4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7" Oct 30 19:14:34.524550 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.524518 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c4r8\" (UniqueName: \"kubernetes.io/projected/af10d82c-0a5c-4139-958c-47721326cf4c-kube-api-access-2c4r8\") pod \"maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7\" (UID: \"af10d82c-0a5c-4139-958c-47721326cf4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7" Oct 30 19:14:34.524646 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.524629 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af10d82c-0a5c-4139-958c-47721326cf4c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7\" (UID: \"af10d82c-0a5c-4139-958c-47721326cf4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7" Oct 30 19:14:34.529769 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.529734 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:14:34.529956 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.529806 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:14:34.529956 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.529827 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:14:34.529956 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.529889 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:14:34.532698 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.532670 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7"] Oct 30 19:14:34.625622 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.625585 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c4r8\" (UniqueName: \"kubernetes.io/projected/af10d82c-0a5c-4139-958c-47721326cf4c-kube-api-access-2c4r8\") pod \"maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7\" (UID: \"af10d82c-0a5c-4139-958c-47721326cf4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7" Oct 30 19:14:34.625821 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.625665 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af10d82c-0a5c-4139-958c-47721326cf4c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7\" (UID: \"af10d82c-0a5c-4139-958c-47721326cf4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7" Oct 30 19:14:34.625821 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.625759 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af10d82c-0a5c-4139-958c-47721326cf4c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7\" (UID: \"af10d82c-0a5c-4139-958c-47721326cf4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7" Oct 30 19:14:34.626404 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.626377 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af10d82c-0a5c-4139-958c-47721326cf4c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7\" (UID: \"af10d82c-0a5c-4139-958c-47721326cf4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7" Oct 30 19:14:34.628426 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.628401 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af10d82c-0a5c-4139-958c-47721326cf4c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7\" (UID: \"af10d82c-0a5c-4139-958c-47721326cf4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7" Oct 30 19:14:34.644224 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.644186 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:14:34.689404 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.689360 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c4r8\" (UniqueName: \"kubernetes.io/projected/af10d82c-0a5c-4139-958c-47721326cf4c-kube-api-access-2c4r8\") pod \"maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7\" (UID: \"af10d82c-0a5c-4139-958c-47721326cf4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7" Oct 30 19:14:34.726412 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.726372 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-credential-socket\") pod \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " Oct 30 19:14:34.726618 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.726424 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-data\") pod \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " Oct 30 19:14:34.726618 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.726457 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-token\" (UniqueName: \"kubernetes.io/projected/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-token\") pod \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " Oct 30 19:14:34.726618 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.726510 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-workload-socket\") pod \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " Oct 30 19:14:34.726618 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.726532 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-workload-certs\") pod \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " Oct 30 19:14:34.726618 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.726570 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-podinfo\") pod \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " Oct 30 19:14:34.726618 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.726596 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-envoy\") pod \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " Oct 30 19:14:34.726946 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.726697 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istiod-ca-cert\") pod \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " Oct 30 19:14:34.726946 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.726716 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cac0c9e8-98a5-49b1-93c4-944bfda920a2-kube-api-access-zpsgp\" (UniqueName: \"kubernetes.io/projected/cac0c9e8-98a5-49b1-93c4-944bfda920a2-kube-api-access-zpsgp\") pod \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\" (UID: \"cac0c9e8-98a5-49b1-93c4-944bfda920a2\") " Oct 30 19:14:34.726946 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.726769 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-data" pod "cac0c9e8-98a5-49b1-93c4-944bfda920a2" (UID: "cac0c9e8-98a5-49b1-93c4-944bfda920a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:34.726946 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.726926 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-workload-socket" pod "cac0c9e8-98a5-49b1-93c4-944bfda920a2" (UID: "cac0c9e8-98a5-49b1-93c4-944bfda920a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:34.726946 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.726939 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-workload-certs" pod "cac0c9e8-98a5-49b1-93c4-944bfda920a2" (UID: "cac0c9e8-98a5-49b1-93c4-944bfda920a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:34.727247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.726732 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-credential-socket" pod "cac0c9e8-98a5-49b1-93c4-944bfda920a2" (UID: "cac0c9e8-98a5-49b1-93c4-944bfda920a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:34.727247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.727097 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istiod-ca-cert" pod "cac0c9e8-98a5-49b1-93c4-944bfda920a2" (UID: "cac0c9e8-98a5-49b1-93c4-944bfda920a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:34.727247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.727189 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:14:34.727247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.727211 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:14:34.727247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.727226 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:14:34.728996 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.728963 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-podinfo" pod "cac0c9e8-98a5-49b1-93c4-944bfda920a2" (UID: "cac0c9e8-98a5-49b1-93c4-944bfda920a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:14:34.729119 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.729016 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-token" pod "cac0c9e8-98a5-49b1-93c4-944bfda920a2" (UID: "cac0c9e8-98a5-49b1-93c4-944bfda920a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:34.729207 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.729118 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-envoy" pod "cac0c9e8-98a5-49b1-93c4-944bfda920a2" (UID: "cac0c9e8-98a5-49b1-93c4-944bfda920a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:34.729341 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.729322 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac0c9e8-98a5-49b1-93c4-944bfda920a2-kube-api-access-zpsgp" pod "cac0c9e8-98a5-49b1-93c4-944bfda920a2" (UID: "cac0c9e8-98a5-49b1-93c4-944bfda920a2"). InnerVolumeSpecName "kube-api-access-zpsgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:34.741334 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.741303 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7" Oct 30 19:14:34.828114 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.828076 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:14:34.828114 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.828110 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:14:34.828114 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.828124 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpsgp\" (UniqueName: \"kubernetes.io/projected/cac0c9e8-98a5-49b1-93c4-944bfda920a2-kube-api-access-zpsgp\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:14:34.828114 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.828155 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac0c9e8-98a5-49b1-93c4-944bfda920a2-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:14:34.828433 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.828168 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:14:34.828433 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:34.828179 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cac0c9e8-98a5-49b1-93c4-944bfda920a2-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:14:35.005046 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:35.005005 4531 generic.go:356] "Generic (PLEG): container finished" podID="cac0c9e8-98a5-49b1-93c4-944bfda920a2" containerID="c6f6e39e85a67bdf500076d74258c06679610567828a33329ce5a2f251ca9030" exitCode=0 Oct 30 19:14:35.005250 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:35.005066 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" Oct 30 19:14:35.005250 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:35.005079 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" event={"ID":"cac0c9e8-98a5-49b1-93c4-944bfda920a2","Type":"ContainerDied","Data":"c6f6e39e85a67bdf500076d74258c06679610567828a33329ce5a2f251ca9030"} Oct 30 19:14:35.005250 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:35.005174 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c" event={"ID":"cac0c9e8-98a5-49b1-93c4-944bfda920a2","Type":"ContainerDied","Data":"1c37c7e0d6f9e7cab10528776008804284d3dfa92bb47e33cdb6cc889bcfe7a7"} Oct 30 19:14:35.005250 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:35.005199 4531 scope.go:122] "RemoveContainer" containerID="c6f6e39e85a67bdf500076d74258c06679610567828a33329ce5a2f251ca9030" Oct 30 19:14:35.016045 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:35.016024 4531 scope.go:122] "RemoveContainer" containerID="132f38aa4f207a572b9c55dbf6123e70927c6484a46538a435913fec0cb0b204" Oct 30 19:14:35.028726 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:35.028698 4531 scope.go:122] "RemoveContainer" containerID="c6f6e39e85a67bdf500076d74258c06679610567828a33329ce5a2f251ca9030" Oct 30 19:14:35.029189 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:14:35.029163 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6f6e39e85a67bdf500076d74258c06679610567828a33329ce5a2f251ca9030\": container with ID starting with c6f6e39e85a67bdf500076d74258c06679610567828a33329ce5a2f251ca9030 not found: ID does not exist" containerID="c6f6e39e85a67bdf500076d74258c06679610567828a33329ce5a2f251ca9030" Oct 30 19:14:35.029266 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:35.029208 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6f6e39e85a67bdf500076d74258c06679610567828a33329ce5a2f251ca9030"} err="failed to get container status \"c6f6e39e85a67bdf500076d74258c06679610567828a33329ce5a2f251ca9030\": rpc error: code = NotFound desc = could not find container \"c6f6e39e85a67bdf500076d74258c06679610567828a33329ce5a2f251ca9030\": container with ID starting with c6f6e39e85a67bdf500076d74258c06679610567828a33329ce5a2f251ca9030 not found: ID does not exist" Oct 30 19:14:35.029266 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:35.029231 4531 scope.go:122] "RemoveContainer" containerID="132f38aa4f207a572b9c55dbf6123e70927c6484a46538a435913fec0cb0b204" Oct 30 19:14:35.029548 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:14:35.029528 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"132f38aa4f207a572b9c55dbf6123e70927c6484a46538a435913fec0cb0b204\": container with ID starting with 132f38aa4f207a572b9c55dbf6123e70927c6484a46538a435913fec0cb0b204 not found: ID does not exist" containerID="132f38aa4f207a572b9c55dbf6123e70927c6484a46538a435913fec0cb0b204" Oct 30 19:14:35.029596 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:35.029556 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"132f38aa4f207a572b9c55dbf6123e70927c6484a46538a435913fec0cb0b204"} err="failed to get container status \"132f38aa4f207a572b9c55dbf6123e70927c6484a46538a435913fec0cb0b204\": rpc error: code = NotFound desc = could not find container \"132f38aa4f207a572b9c55dbf6123e70927c6484a46538a435913fec0cb0b204\": container with ID starting with 132f38aa4f207a572b9c55dbf6123e70927c6484a46538a435913fec0cb0b204 not found: ID does not exist" Oct 30 19:14:35.134604 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:35.134578 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7"] Oct 30 19:14:35.137590 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:14:35.137564 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf10d82c_0a5c_4139_958c_47721326cf4c.slice/crio-dd9c4b4bc717d83f5ef64b378b26348ea736fcdb02d15cf02aa3427a97875c8d WatchSource:0}: Error finding container dd9c4b4bc717d83f5ef64b378b26348ea736fcdb02d15cf02aa3427a97875c8d: Status 404 returned error can't find the container with id dd9c4b4bc717d83f5ef64b378b26348ea736fcdb02d15cf02aa3427a97875c8d Oct 30 19:14:35.184818 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:35.184787 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c"] Oct 30 19:14:35.195864 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:35.195829 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-5a035d-openshift-default-6ff84fc4b9-qnw4c"] Oct 30 19:14:36.013798 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:36.013757 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7" event={"ID":"af10d82c-0a5c-4139-958c-47721326cf4c","Type":"ContainerStarted","Data":"5c9a9b1006649e886183d231cb44b9e56303652ed5071c7a43ddceb70bd7115d"} Oct 30 19:14:36.014223 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:36.013814 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7" event={"ID":"af10d82c-0a5c-4139-958c-47721326cf4c","Type":"ContainerStarted","Data":"dd9c4b4bc717d83f5ef64b378b26348ea736fcdb02d15cf02aa3427a97875c8d"} Oct 30 19:14:36.098826 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:36.098789 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cac0c9e8-98a5-49b1-93c4-944bfda920a2" path="/var/lib/kubelet/pods/cac0c9e8-98a5-49b1-93c4-944bfda920a2/volumes" Oct 30 19:14:36.132503 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:36.132438 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7" podStartSLOduration=2.132420201 podStartE2EDuration="2.132420201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:36.130169365 +0000 UTC m=+2436.157274326" watchObservedRunningTime="2026-10-30 19:14:36.132420201 +0000 UTC m=+2436.159525160" Oct 30 19:14:37.001638 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:37.001601 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-575cced8-2zvz6"] Oct 30 19:14:37.002083 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:37.002070 4531 cpu_manager.go:414] "containerMap: removing container" podUID="cac0c9e8-98a5-49b1-93c4-944bfda920a2" containerName="istio-proxy" Oct 30 19:14:37.002171 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:37.002085 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cac0c9e8-98a5-49b1-93c4-944bfda920a2" containerName="istio-proxy" Oct 30 19:14:37.002217 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:37.002182 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="cac0c9e8-98a5-49b1-93c4-944bfda920a2" containerName="istio-proxy" Oct 30 19:14:37.002217 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:37.002193 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="cac0c9e8-98a5-49b1-93c4-944bfda920a2" containerName="istio-proxy" Oct 30 19:14:37.005301 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:37.005284 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575cced8-2zvz6" Oct 30 19:14:37.021046 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:37.021005 4531 generic.go:356] "Generic (PLEG): container finished" podID="af10d82c-0a5c-4139-958c-47721326cf4c" containerID="5c9a9b1006649e886183d231cb44b9e56303652ed5071c7a43ddceb70bd7115d" exitCode=0 Oct 30 19:14:37.021438 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:37.021056 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7" event={"ID":"af10d82c-0a5c-4139-958c-47721326cf4c","Type":"ContainerDied","Data":"5c9a9b1006649e886183d231cb44b9e56303652ed5071c7a43ddceb70bd7115d"} Oct 30 19:14:37.052854 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:37.052811 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff7wl\" (UniqueName: \"kubernetes.io/projected/9ed85520-ff89-44e0-a03e-8aa423fe3cf5-kube-api-access-ff7wl\") pod \"maas-api-revoke-sub-575cced8-2zvz6\" (UID: \"9ed85520-ff89-44e0-a03e-8aa423fe3cf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575cced8-2zvz6" Oct 30 19:14:37.053052 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:37.052926 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed85520-ff89-44e0-a03e-8aa423fe3cf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-575cced8-2zvz6\" (UID: \"9ed85520-ff89-44e0-a03e-8aa423fe3cf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575cced8-2zvz6" Oct 30 19:14:37.053052 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:37.052955 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed85520-ff89-44e0-a03e-8aa423fe3cf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-575cced8-2zvz6\" (UID: \"9ed85520-ff89-44e0-a03e-8aa423fe3cf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575cced8-2zvz6" Oct 30 19:14:37.064670 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:37.064639 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-575cced8-2zvz6"] Oct 30 19:14:37.154661 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:37.154618 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed85520-ff89-44e0-a03e-8aa423fe3cf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-575cced8-2zvz6\" (UID: \"9ed85520-ff89-44e0-a03e-8aa423fe3cf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575cced8-2zvz6" Oct 30 19:14:37.154856 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:37.154752 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff7wl\" (UniqueName: \"kubernetes.io/projected/9ed85520-ff89-44e0-a03e-8aa423fe3cf5-kube-api-access-ff7wl\") pod \"maas-api-revoke-sub-575cced8-2zvz6\" (UID: \"9ed85520-ff89-44e0-a03e-8aa423fe3cf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575cced8-2zvz6" Oct 30 19:14:37.154856 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:37.154824 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed85520-ff89-44e0-a03e-8aa423fe3cf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-575cced8-2zvz6\" (UID: \"9ed85520-ff89-44e0-a03e-8aa423fe3cf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575cced8-2zvz6" Oct 30 19:14:37.155563 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:37.155537 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed85520-ff89-44e0-a03e-8aa423fe3cf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-575cced8-2zvz6\" (UID: \"9ed85520-ff89-44e0-a03e-8aa423fe3cf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575cced8-2zvz6" Oct 30 19:14:37.157398 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:37.157366 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed85520-ff89-44e0-a03e-8aa423fe3cf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-575cced8-2zvz6\" (UID: \"9ed85520-ff89-44e0-a03e-8aa423fe3cf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575cced8-2zvz6" Oct 30 19:14:37.221491 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:37.221454 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff7wl\" (UniqueName: \"kubernetes.io/projected/9ed85520-ff89-44e0-a03e-8aa423fe3cf5-kube-api-access-ff7wl\") pod \"maas-api-revoke-sub-575cced8-2zvz6\" (UID: \"9ed85520-ff89-44e0-a03e-8aa423fe3cf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575cced8-2zvz6" Oct 30 19:14:37.317859 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:37.317763 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575cced8-2zvz6" Oct 30 19:14:37.698611 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:37.698584 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-575cced8-2zvz6"] Oct 30 19:14:37.701318 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:14:37.701282 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed85520_ff89_44e0_a03e_8aa423fe3cf5.slice/crio-1cb0d86d89f1b235dd9621bc918c71cc22895892a9ff2565762d0f9ba94b53f9 WatchSource:0}: Error finding container 1cb0d86d89f1b235dd9621bc918c71cc22895892a9ff2565762d0f9ba94b53f9: Status 404 returned error can't find the container with id 1cb0d86d89f1b235dd9621bc918c71cc22895892a9ff2565762d0f9ba94b53f9 Oct 30 19:14:38.028836 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:38.028798 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575cced8-2zvz6" event={"ID":"9ed85520-ff89-44e0-a03e-8aa423fe3cf5","Type":"ContainerStarted","Data":"1cb0d86d89f1b235dd9621bc918c71cc22895892a9ff2565762d0f9ba94b53f9"} Oct 30 19:14:38.194960 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:38.194927 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7" Oct 30 19:14:38.266379 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:38.266339 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af10d82c-0a5c-4139-958c-47721326cf4c-kube-api-access-2c4r8\" (UniqueName: \"kubernetes.io/projected/af10d82c-0a5c-4139-958c-47721326cf4c-kube-api-access-2c4r8\") pod \"af10d82c-0a5c-4139-958c-47721326cf4c\" (UID: \"af10d82c-0a5c-4139-958c-47721326cf4c\") " Oct 30 19:14:38.266574 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:38.266390 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af10d82c-0a5c-4139-958c-47721326cf4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af10d82c-0a5c-4139-958c-47721326cf4c-maas-api-openshift-ca\") pod \"af10d82c-0a5c-4139-958c-47721326cf4c\" (UID: \"af10d82c-0a5c-4139-958c-47721326cf4c\") " Oct 30 19:14:38.266574 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:38.266430 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af10d82c-0a5c-4139-958c-47721326cf4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af10d82c-0a5c-4139-958c-47721326cf4c-maas-api-xks-ca\") pod \"af10d82c-0a5c-4139-958c-47721326cf4c\" (UID: \"af10d82c-0a5c-4139-958c-47721326cf4c\") " Oct 30 19:14:38.266806 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:38.266777 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af10d82c-0a5c-4139-958c-47721326cf4c-maas-api-openshift-ca" pod "af10d82c-0a5c-4139-958c-47721326cf4c" (UID: "af10d82c-0a5c-4139-958c-47721326cf4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:38.268860 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:38.268825 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af10d82c-0a5c-4139-958c-47721326cf4c-maas-api-xks-ca" pod "af10d82c-0a5c-4139-958c-47721326cf4c" (UID: "af10d82c-0a5c-4139-958c-47721326cf4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:14:38.268860 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:38.268848 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af10d82c-0a5c-4139-958c-47721326cf4c-kube-api-access-2c4r8" pod "af10d82c-0a5c-4139-958c-47721326cf4c" (UID: "af10d82c-0a5c-4139-958c-47721326cf4c"). InnerVolumeSpecName "kube-api-access-2c4r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:38.368018 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:38.367979 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c4r8\" (UniqueName: \"kubernetes.io/projected/af10d82c-0a5c-4139-958c-47721326cf4c-kube-api-access-2c4r8\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:14:38.368018 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:38.368010 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af10d82c-0a5c-4139-958c-47721326cf4c-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:14:38.368018 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:38.368024 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af10d82c-0a5c-4139-958c-47721326cf4c-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:14:39.037221 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:39.037163 4531 generic.go:356] "Generic (PLEG): container finished" podID="9ed85520-ff89-44e0-a03e-8aa423fe3cf5" containerID="8f244f10d01d1dadf4e3c3156a4c9303ca5e59803b69310dfe73e9759a1144c3" exitCode=0 Oct 30 19:14:39.037696 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:39.037233 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575cced8-2zvz6" event={"ID":"9ed85520-ff89-44e0-a03e-8aa423fe3cf5","Type":"ContainerDied","Data":"8f244f10d01d1dadf4e3c3156a4c9303ca5e59803b69310dfe73e9759a1144c3"} Oct 30 19:14:39.038761 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:39.038732 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7" event={"ID":"af10d82c-0a5c-4139-958c-47721326cf4c","Type":"ContainerDied","Data":"dd9c4b4bc717d83f5ef64b378b26348ea736fcdb02d15cf02aa3427a97875c8d"} Oct 30 19:14:39.038761 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:39.038758 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7" Oct 30 19:14:39.038886 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:39.038787 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd9c4b4bc717d83f5ef64b378b26348ea736fcdb02d15cf02aa3427a97875c8d" Oct 30 19:14:40.193423 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:40.193367 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575cced8-2zvz6" Oct 30 19:14:40.290795 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:40.290759 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ed85520-ff89-44e0-a03e-8aa423fe3cf5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed85520-ff89-44e0-a03e-8aa423fe3cf5-maas-api-openshift-ca\") pod \"9ed85520-ff89-44e0-a03e-8aa423fe3cf5\" (UID: \"9ed85520-ff89-44e0-a03e-8aa423fe3cf5\") " Oct 30 19:14:40.290999 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:40.290887 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ed85520-ff89-44e0-a03e-8aa423fe3cf5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed85520-ff89-44e0-a03e-8aa423fe3cf5-maas-api-xks-ca\") pod \"9ed85520-ff89-44e0-a03e-8aa423fe3cf5\" (UID: \"9ed85520-ff89-44e0-a03e-8aa423fe3cf5\") " Oct 30 19:14:40.290999 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:40.290922 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ed85520-ff89-44e0-a03e-8aa423fe3cf5-kube-api-access-ff7wl\" (UniqueName: \"kubernetes.io/projected/9ed85520-ff89-44e0-a03e-8aa423fe3cf5-kube-api-access-ff7wl\") pod \"9ed85520-ff89-44e0-a03e-8aa423fe3cf5\" (UID: \"9ed85520-ff89-44e0-a03e-8aa423fe3cf5\") " Oct 30 19:14:40.291267 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:40.291237 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ed85520-ff89-44e0-a03e-8aa423fe3cf5-maas-api-openshift-ca" pod "9ed85520-ff89-44e0-a03e-8aa423fe3cf5" (UID: "9ed85520-ff89-44e0-a03e-8aa423fe3cf5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:40.293259 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:40.293224 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed85520-ff89-44e0-a03e-8aa423fe3cf5-kube-api-access-ff7wl" pod "9ed85520-ff89-44e0-a03e-8aa423fe3cf5" (UID: "9ed85520-ff89-44e0-a03e-8aa423fe3cf5"). InnerVolumeSpecName "kube-api-access-ff7wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:40.293259 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:40.293249 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ed85520-ff89-44e0-a03e-8aa423fe3cf5-maas-api-xks-ca" pod "9ed85520-ff89-44e0-a03e-8aa423fe3cf5" (UID: "9ed85520-ff89-44e0-a03e-8aa423fe3cf5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:14:40.392424 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:40.392383 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff7wl\" (UniqueName: \"kubernetes.io/projected/9ed85520-ff89-44e0-a03e-8aa423fe3cf5-kube-api-access-ff7wl\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:14:40.392424 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:40.392418 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed85520-ff89-44e0-a03e-8aa423fe3cf5-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:14:40.392424 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:40.392430 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed85520-ff89-44e0-a03e-8aa423fe3cf5-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:14:41.051083 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:41.051049 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575cced8-2zvz6" Oct 30 19:14:41.051282 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:41.051082 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575cced8-2zvz6" event={"ID":"9ed85520-ff89-44e0-a03e-8aa423fe3cf5","Type":"ContainerDied","Data":"1cb0d86d89f1b235dd9621bc918c71cc22895892a9ff2565762d0f9ba94b53f9"} Oct 30 19:14:41.051282 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:41.051155 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cb0d86d89f1b235dd9621bc918c71cc22895892a9ff2565762d0f9ba94b53f9" Oct 30 19:14:45.927689 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:45.927654 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5"] Oct 30 19:14:46.960052 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:46.960015 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6d3eb80-n9c2t"] Oct 30 19:14:46.960523 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:46.960508 4531 cpu_manager.go:414] "containerMap: removing container" podUID="9ed85520-ff89-44e0-a03e-8aa423fe3cf5" containerName="revoke-keys" Oct 30 19:14:46.960578 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:46.960525 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ed85520-ff89-44e0-a03e-8aa423fe3cf5" containerName="revoke-keys" Oct 30 19:14:46.960578 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:46.960543 4531 cpu_manager.go:414] "containerMap: removing container" podUID="af10d82c-0a5c-4139-958c-47721326cf4c" containerName="revoke-keys" Oct 30 19:14:46.960578 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:46.960548 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af10d82c-0a5c-4139-958c-47721326cf4c" containerName="revoke-keys" Oct 30 19:14:46.960578 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:46.960567 4531 cpu_manager.go:414] "containerMap: removing container" podUID="cac0c9e8-98a5-49b1-93c4-944bfda920a2" containerName="istio-proxy" Oct 30 19:14:46.960578 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:46.960572 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cac0c9e8-98a5-49b1-93c4-944bfda920a2" containerName="istio-proxy" Oct 30 19:14:46.960758 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:46.960647 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ed85520-ff89-44e0-a03e-8aa423fe3cf5" containerName="revoke-keys" Oct 30 19:14:46.960758 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:46.960657 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="af10d82c-0a5c-4139-958c-47721326cf4c" containerName="revoke-keys" Oct 30 19:14:46.963775 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:46.963757 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d3eb80-n9c2t" Oct 30 19:14:47.016227 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:47.016194 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:14:47.016411 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:47.016227 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:14:47.016411 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:47.016276 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:14:47.016411 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:47.016305 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:14:47.022778 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:47.022741 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6d3eb80-n9c2t"] Oct 30 19:14:47.068403 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:47.068362 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84ef0d55-d199-4b86-8113-36bbbc61f614-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6d3eb80-n9c2t\" (UID: \"84ef0d55-d199-4b86-8113-36bbbc61f614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d3eb80-n9c2t" Oct 30 19:14:47.068627 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:47.068457 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rhdn\" (UniqueName: \"kubernetes.io/projected/84ef0d55-d199-4b86-8113-36bbbc61f614-kube-api-access-8rhdn\") pod \"maas-api-revoke-sub-f6d3eb80-n9c2t\" (UID: \"84ef0d55-d199-4b86-8113-36bbbc61f614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d3eb80-n9c2t" Oct 30 19:14:47.068707 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:47.068638 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84ef0d55-d199-4b86-8113-36bbbc61f614-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6d3eb80-n9c2t\" (UID: \"84ef0d55-d199-4b86-8113-36bbbc61f614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d3eb80-n9c2t" Oct 30 19:14:47.169684 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:47.169651 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84ef0d55-d199-4b86-8113-36bbbc61f614-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6d3eb80-n9c2t\" (UID: \"84ef0d55-d199-4b86-8113-36bbbc61f614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d3eb80-n9c2t" Oct 30 19:14:47.169889 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:47.169718 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rhdn\" (UniqueName: \"kubernetes.io/projected/84ef0d55-d199-4b86-8113-36bbbc61f614-kube-api-access-8rhdn\") pod \"maas-api-revoke-sub-f6d3eb80-n9c2t\" (UID: \"84ef0d55-d199-4b86-8113-36bbbc61f614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d3eb80-n9c2t" Oct 30 19:14:47.169889 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:47.169872 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84ef0d55-d199-4b86-8113-36bbbc61f614-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6d3eb80-n9c2t\" (UID: \"84ef0d55-d199-4b86-8113-36bbbc61f614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d3eb80-n9c2t" Oct 30 19:14:47.170412 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:47.170390 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84ef0d55-d199-4b86-8113-36bbbc61f614-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6d3eb80-n9c2t\" (UID: \"84ef0d55-d199-4b86-8113-36bbbc61f614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d3eb80-n9c2t" Oct 30 19:14:47.172338 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:47.172308 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84ef0d55-d199-4b86-8113-36bbbc61f614-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6d3eb80-n9c2t\" (UID: \"84ef0d55-d199-4b86-8113-36bbbc61f614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d3eb80-n9c2t" Oct 30 19:14:47.231055 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:47.230976 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rhdn\" (UniqueName: \"kubernetes.io/projected/84ef0d55-d199-4b86-8113-36bbbc61f614-kube-api-access-8rhdn\") pod \"maas-api-revoke-sub-f6d3eb80-n9c2t\" (UID: \"84ef0d55-d199-4b86-8113-36bbbc61f614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d3eb80-n9c2t" Oct 30 19:14:47.277345 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:47.277301 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d3eb80-n9c2t" Oct 30 19:14:47.662098 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:47.662061 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6d3eb80-n9c2t"] Oct 30 19:14:48.097654 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:48.097609 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d3eb80-n9c2t" event={"ID":"84ef0d55-d199-4b86-8113-36bbbc61f614","Type":"ContainerStarted","Data":"f3d1648489d1526df34dd9e2a236b77323c9d64eb91a4f7919be15dce7c74104"} Oct 30 19:14:48.098050 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:48.097675 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d3eb80-n9c2t" event={"ID":"84ef0d55-d199-4b86-8113-36bbbc61f614","Type":"ContainerStarted","Data":"8b91df88c1238df5eb162b109b2badebbbb5d5a459c7977090fa44c51fa5ecd7"} Oct 30 19:14:48.208881 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:48.208769 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d3eb80-n9c2t" podStartSLOduration=2.208753452 podStartE2EDuration="2.208753452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:48.207330391 +0000 UTC m=+2448.234435352" watchObservedRunningTime="2026-10-30 19:14:48.208753452 +0000 UTC m=+2448.235858411" Oct 30 19:14:49.098555 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:49.098521 4531 generic.go:356] "Generic (PLEG): container finished" podID="84ef0d55-d199-4b86-8113-36bbbc61f614" containerID="f3d1648489d1526df34dd9e2a236b77323c9d64eb91a4f7919be15dce7c74104" exitCode=0 Oct 30 19:14:49.098960 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:49.098595 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d3eb80-n9c2t" event={"ID":"84ef0d55-d199-4b86-8113-36bbbc61f614","Type":"ContainerDied","Data":"f3d1648489d1526df34dd9e2a236b77323c9d64eb91a4f7919be15dce7c74104"} Oct 30 19:14:50.254846 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:50.254812 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d3eb80-n9c2t" Oct 30 19:14:50.405555 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:50.405457 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84ef0d55-d199-4b86-8113-36bbbc61f614-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84ef0d55-d199-4b86-8113-36bbbc61f614-maas-api-xks-ca\") pod \"84ef0d55-d199-4b86-8113-36bbbc61f614\" (UID: \"84ef0d55-d199-4b86-8113-36bbbc61f614\") " Oct 30 19:14:50.405759 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:50.405583 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84ef0d55-d199-4b86-8113-36bbbc61f614-kube-api-access-8rhdn\" (UniqueName: \"kubernetes.io/projected/84ef0d55-d199-4b86-8113-36bbbc61f614-kube-api-access-8rhdn\") pod \"84ef0d55-d199-4b86-8113-36bbbc61f614\" (UID: \"84ef0d55-d199-4b86-8113-36bbbc61f614\") " Oct 30 19:14:50.405759 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:50.405702 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84ef0d55-d199-4b86-8113-36bbbc61f614-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84ef0d55-d199-4b86-8113-36bbbc61f614-maas-api-openshift-ca\") pod \"84ef0d55-d199-4b86-8113-36bbbc61f614\" (UID: \"84ef0d55-d199-4b86-8113-36bbbc61f614\") " Oct 30 19:14:50.406253 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:50.406227 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84ef0d55-d199-4b86-8113-36bbbc61f614-maas-api-openshift-ca" pod "84ef0d55-d199-4b86-8113-36bbbc61f614" (UID: "84ef0d55-d199-4b86-8113-36bbbc61f614"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:50.406470 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:50.406453 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84ef0d55-d199-4b86-8113-36bbbc61f614-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.411487 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:50.411455 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84ef0d55-d199-4b86-8113-36bbbc61f614-maas-api-xks-ca" pod "84ef0d55-d199-4b86-8113-36bbbc61f614" (UID: "84ef0d55-d199-4b86-8113-36bbbc61f614"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:14:50.411608 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:50.411587 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84ef0d55-d199-4b86-8113-36bbbc61f614-kube-api-access-8rhdn" pod "84ef0d55-d199-4b86-8113-36bbbc61f614" (UID: "84ef0d55-d199-4b86-8113-36bbbc61f614"). InnerVolumeSpecName "kube-api-access-8rhdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:50.508063 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:50.508027 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rhdn\" (UniqueName: \"kubernetes.io/projected/84ef0d55-d199-4b86-8113-36bbbc61f614-kube-api-access-8rhdn\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.508063 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:50.508059 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84ef0d55-d199-4b86-8113-36bbbc61f614-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:14:50.928868 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:50.928826 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5" podUID="97759a69-ea8f-4f5d-b419-a6c314b0c47d" containerName="payload-processing" containerID="cri-o://df5dd55b5b0e3e86fc9fc8a682d34bd4e0c9f7edebd3e76db42f2d13119fe5c1" gracePeriod=25 Oct 30 19:14:51.115100 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:51.115069 4531 generic.go:356] "Generic (PLEG): container finished" podID="97759a69-ea8f-4f5d-b419-a6c314b0c47d" containerID="df5dd55b5b0e3e86fc9fc8a682d34bd4e0c9f7edebd3e76db42f2d13119fe5c1" exitCode=0 Oct 30 19:14:51.115310 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:51.115176 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5" event={"ID":"97759a69-ea8f-4f5d-b419-a6c314b0c47d","Type":"ContainerDied","Data":"df5dd55b5b0e3e86fc9fc8a682d34bd4e0c9f7edebd3e76db42f2d13119fe5c1"} Oct 30 19:14:51.116777 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:51.116748 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d3eb80-n9c2t" event={"ID":"84ef0d55-d199-4b86-8113-36bbbc61f614","Type":"ContainerDied","Data":"8b91df88c1238df5eb162b109b2badebbbb5d5a459c7977090fa44c51fa5ecd7"} Oct 30 19:14:51.116899 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:51.116799 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b91df88c1238df5eb162b109b2badebbbb5d5a459c7977090fa44c51fa5ecd7" Oct 30 19:14:51.116899 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:51.116816 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d3eb80-n9c2t" Oct 30 19:14:51.185545 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:51.185482 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5" Oct 30 19:14:51.315541 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:51.315504 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97759a69-ea8f-4f5d-b419-a6c314b0c47d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97759a69-ea8f-4f5d-b419-a6c314b0c47d-plugins-config-volume\") pod \"97759a69-ea8f-4f5d-b419-a6c314b0c47d\" (UID: \"97759a69-ea8f-4f5d-b419-a6c314b0c47d\") " Oct 30 19:14:51.315971 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:51.315576 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97759a69-ea8f-4f5d-b419-a6c314b0c47d-kube-api-access-79nlb\" (UniqueName: \"kubernetes.io/projected/97759a69-ea8f-4f5d-b419-a6c314b0c47d-kube-api-access-79nlb\") pod \"97759a69-ea8f-4f5d-b419-a6c314b0c47d\" (UID: \"97759a69-ea8f-4f5d-b419-a6c314b0c47d\") " Oct 30 19:14:51.316022 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:51.315960 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97759a69-ea8f-4f5d-b419-a6c314b0c47d-plugins-config-volume" pod "97759a69-ea8f-4f5d-b419-a6c314b0c47d" (UID: "97759a69-ea8f-4f5d-b419-a6c314b0c47d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:51.316171 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:51.316148 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97759a69-ea8f-4f5d-b419-a6c314b0c47d-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:14:51.318392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:51.318361 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97759a69-ea8f-4f5d-b419-a6c314b0c47d-kube-api-access-79nlb" pod "97759a69-ea8f-4f5d-b419-a6c314b0c47d" (UID: "97759a69-ea8f-4f5d-b419-a6c314b0c47d"). InnerVolumeSpecName "kube-api-access-79nlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:51.417762 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:51.417699 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79nlb\" (UniqueName: \"kubernetes.io/projected/97759a69-ea8f-4f5d-b419-a6c314b0c47d-kube-api-access-79nlb\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:14:52.123276 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:52.123241 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5" event={"ID":"97759a69-ea8f-4f5d-b419-a6c314b0c47d","Type":"ContainerDied","Data":"048e904fd5c7fdc26d91b1c3d2ece32c7466ece36839a1b325cbfb4c71d88d4b"} Oct 30 19:14:52.123482 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:52.123310 4531 scope.go:122] "RemoveContainer" containerID="df5dd55b5b0e3e86fc9fc8a682d34bd4e0c9f7edebd3e76db42f2d13119fe5c1" Oct 30 19:14:52.123482 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:52.123248 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5" Oct 30 19:14:52.287417 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:52.287380 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5"] Oct 30 19:14:52.337007 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:52.336972 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0be4a4-59c599c668-kkbc5"] Oct 30 19:14:54.098070 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:54.098033 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97759a69-ea8f-4f5d-b419-a6c314b0c47d" path="/var/lib/kubelet/pods/97759a69-ea8f-4f5d-b419-a6c314b0c47d/volumes" Oct 30 19:14:55.550009 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:55.549971 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7064831-hpp87"] Oct 30 19:14:55.550456 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:55.550441 4531 cpu_manager.go:414] "containerMap: removing container" podUID="84ef0d55-d199-4b86-8113-36bbbc61f614" containerName="revoke-keys" Oct 30 19:14:55.550514 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:55.550457 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84ef0d55-d199-4b86-8113-36bbbc61f614" containerName="revoke-keys" Oct 30 19:14:55.550514 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:55.550492 4531 cpu_manager.go:414] "containerMap: removing container" podUID="97759a69-ea8f-4f5d-b419-a6c314b0c47d" containerName="payload-processing" Oct 30 19:14:55.550514 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:55.550497 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97759a69-ea8f-4f5d-b419-a6c314b0c47d" containerName="payload-processing" Oct 30 19:14:55.550612 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:55.550565 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="84ef0d55-d199-4b86-8113-36bbbc61f614" containerName="revoke-keys" Oct 30 19:14:55.550612 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:55.550581 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="97759a69-ea8f-4f5d-b419-a6c314b0c47d" containerName="payload-processing" Oct 30 19:14:55.555241 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:55.555224 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7064831-hpp87" Oct 30 19:14:55.610389 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:55.610363 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:14:55.610595 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:55.610491 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:14:55.612884 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:55.612861 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7064831-hpp87"] Oct 30 19:14:55.658908 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:55.658881 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:14:55.659048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:55.658954 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:14:55.666201 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:55.666179 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d486a241-c368-4e95-ad58-e06f5e41ef24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7064831-hpp87\" (UID: \"d486a241-c368-4e95-ad58-e06f5e41ef24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7064831-hpp87" Oct 30 19:14:55.666370 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:55.666300 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfv99\" (UniqueName: \"kubernetes.io/projected/d486a241-c368-4e95-ad58-e06f5e41ef24-kube-api-access-zfv99\") pod \"maas-api-revoke-sub-b7064831-hpp87\" (UID: \"d486a241-c368-4e95-ad58-e06f5e41ef24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7064831-hpp87" Oct 30 19:14:55.666370 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:55.666344 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d486a241-c368-4e95-ad58-e06f5e41ef24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7064831-hpp87\" (UID: \"d486a241-c368-4e95-ad58-e06f5e41ef24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7064831-hpp87" Oct 30 19:14:55.767701 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:55.767673 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfv99\" (UniqueName: \"kubernetes.io/projected/d486a241-c368-4e95-ad58-e06f5e41ef24-kube-api-access-zfv99\") pod \"maas-api-revoke-sub-b7064831-hpp87\" (UID: \"d486a241-c368-4e95-ad58-e06f5e41ef24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7064831-hpp87" Oct 30 19:14:55.767847 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:55.767714 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d486a241-c368-4e95-ad58-e06f5e41ef24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7064831-hpp87\" (UID: \"d486a241-c368-4e95-ad58-e06f5e41ef24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7064831-hpp87" Oct 30 19:14:55.767847 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:55.767785 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d486a241-c368-4e95-ad58-e06f5e41ef24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7064831-hpp87\" (UID: \"d486a241-c368-4e95-ad58-e06f5e41ef24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7064831-hpp87" Oct 30 19:14:55.768359 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:55.768339 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d486a241-c368-4e95-ad58-e06f5e41ef24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7064831-hpp87\" (UID: \"d486a241-c368-4e95-ad58-e06f5e41ef24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7064831-hpp87" Oct 30 19:14:55.770345 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:55.770321 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d486a241-c368-4e95-ad58-e06f5e41ef24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7064831-hpp87\" (UID: \"d486a241-c368-4e95-ad58-e06f5e41ef24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7064831-hpp87" Oct 30 19:14:55.829630 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:55.829564 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfv99\" (UniqueName: \"kubernetes.io/projected/d486a241-c368-4e95-ad58-e06f5e41ef24-kube-api-access-zfv99\") pod \"maas-api-revoke-sub-b7064831-hpp87\" (UID: \"d486a241-c368-4e95-ad58-e06f5e41ef24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7064831-hpp87" Oct 30 19:14:55.870520 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:55.870494 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7064831-hpp87" Oct 30 19:14:56.250413 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:56.250381 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7064831-hpp87"] Oct 30 19:14:56.251934 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:14:56.251903 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd486a241_c368_4e95_ad58_e06f5e41ef24.slice/crio-75f615278c79193d3851272abec6c55c4405d4895a51e70916facd11cdc811e9 WatchSource:0}: Error finding container 75f615278c79193d3851272abec6c55c4405d4895a51e70916facd11cdc811e9: Status 404 returned error can't find the container with id 75f615278c79193d3851272abec6c55c4405d4895a51e70916facd11cdc811e9 Oct 30 19:14:56.397016 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:14:56.396983 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:14:57.155286 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:57.155260 4531 generic.go:356] "Generic (PLEG): container finished" podID="d486a241-c368-4e95-ad58-e06f5e41ef24" containerID="9625943d87b168c88c650fb74519a2adfe0df6456ce58722ef3e1c0546d21f5d" exitCode=0 Oct 30 19:14:57.155658 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:57.155300 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7064831-hpp87" event={"ID":"d486a241-c368-4e95-ad58-e06f5e41ef24","Type":"ContainerDied","Data":"9625943d87b168c88c650fb74519a2adfe0df6456ce58722ef3e1c0546d21f5d"} Oct 30 19:14:57.155658 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:57.155340 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7064831-hpp87" event={"ID":"d486a241-c368-4e95-ad58-e06f5e41ef24","Type":"ContainerStarted","Data":"75f615278c79193d3851272abec6c55c4405d4895a51e70916facd11cdc811e9"} Oct 30 19:14:58.305191 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:58.305166 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7064831-hpp87" Oct 30 19:14:58.493654 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:58.493632 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d486a241-c368-4e95-ad58-e06f5e41ef24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d486a241-c368-4e95-ad58-e06f5e41ef24-maas-api-xks-ca\") pod \"d486a241-c368-4e95-ad58-e06f5e41ef24\" (UID: \"d486a241-c368-4e95-ad58-e06f5e41ef24\") " Oct 30 19:14:58.493828 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:58.493731 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d486a241-c368-4e95-ad58-e06f5e41ef24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d486a241-c368-4e95-ad58-e06f5e41ef24-maas-api-openshift-ca\") pod \"d486a241-c368-4e95-ad58-e06f5e41ef24\" (UID: \"d486a241-c368-4e95-ad58-e06f5e41ef24\") " Oct 30 19:14:58.493828 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:58.493753 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d486a241-c368-4e95-ad58-e06f5e41ef24-kube-api-access-zfv99\" (UniqueName: \"kubernetes.io/projected/d486a241-c368-4e95-ad58-e06f5e41ef24-kube-api-access-zfv99\") pod \"d486a241-c368-4e95-ad58-e06f5e41ef24\" (UID: \"d486a241-c368-4e95-ad58-e06f5e41ef24\") " Oct 30 19:14:58.494145 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:58.494105 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d486a241-c368-4e95-ad58-e06f5e41ef24-maas-api-openshift-ca" pod "d486a241-c368-4e95-ad58-e06f5e41ef24" (UID: "d486a241-c368-4e95-ad58-e06f5e41ef24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:58.495959 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:58.495932 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d486a241-c368-4e95-ad58-e06f5e41ef24-maas-api-xks-ca" pod "d486a241-c368-4e95-ad58-e06f5e41ef24" (UID: "d486a241-c368-4e95-ad58-e06f5e41ef24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:14:58.496057 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:58.495987 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d486a241-c368-4e95-ad58-e06f5e41ef24-kube-api-access-zfv99" pod "d486a241-c368-4e95-ad58-e06f5e41ef24" (UID: "d486a241-c368-4e95-ad58-e06f5e41ef24"). InnerVolumeSpecName "kube-api-access-zfv99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:58.594794 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:58.594771 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d486a241-c368-4e95-ad58-e06f5e41ef24-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:14:58.594794 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:58.594794 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfv99\" (UniqueName: \"kubernetes.io/projected/d486a241-c368-4e95-ad58-e06f5e41ef24-kube-api-access-zfv99\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:14:58.594916 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:58.594805 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d486a241-c368-4e95-ad58-e06f5e41ef24-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:14:59.166838 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:59.166808 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7064831-hpp87" Oct 30 19:14:59.167010 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:59.166847 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7064831-hpp87" event={"ID":"d486a241-c368-4e95-ad58-e06f5e41ef24","Type":"ContainerDied","Data":"75f615278c79193d3851272abec6c55c4405d4895a51e70916facd11cdc811e9"} Oct 30 19:14:59.167010 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:14:59.166893 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75f615278c79193d3851272abec6c55c4405d4895a51e70916facd11cdc811e9" Oct 30 19:15:00.160634 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.160605 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs"] Oct 30 19:15:00.161047 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.161034 4531 cpu_manager.go:414] "containerMap: removing container" podUID="d486a241-c368-4e95-ad58-e06f5e41ef24" containerName="revoke-keys" Oct 30 19:15:00.161113 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.161049 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d486a241-c368-4e95-ad58-e06f5e41ef24" containerName="revoke-keys" Oct 30 19:15:00.161175 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.161158 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d486a241-c368-4e95-ad58-e06f5e41ef24" containerName="revoke-keys" Oct 30 19:15:00.164386 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.164370 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs" Oct 30 19:15:00.171657 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.171635 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t"] Oct 30 19:15:00.175861 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.175842 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24"] Oct 30 19:15:00.175973 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.175955 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t" Oct 30 19:15:00.178932 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.178916 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24" Oct 30 19:15:00.209944 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.209918 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p"] Oct 30 19:15:00.213811 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.213796 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh"] Oct 30 19:15:00.213918 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.213905 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p" Oct 30 19:15:00.217169 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.217153 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh" Oct 30 19:15:00.219818 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.219803 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2"] Oct 30 19:15:00.221266 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.221245 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:15:00.223394 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.223376 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x"] Oct 30 19:15:00.223508 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.223491 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2" Oct 30 19:15:00.226626 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.226610 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x" Oct 30 19:15:00.227777 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.227761 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:15:00.258037 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.258017 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:15:00.259982 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.259965 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n"] Oct 30 19:15:00.263381 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.263367 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n" Oct 30 19:15:00.270045 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.270025 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p"] Oct 30 19:15:00.273907 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.273889 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs"] Oct 30 19:15:00.275813 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.275797 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t"] Oct 30 19:15:00.277655 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.277639 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24"] Oct 30 19:15:00.312668 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.312626 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrqb8\" (UniqueName: \"kubernetes.io/projected/48841348-a856-4dcc-9687-b987faf399cd-kube-api-access-wrqb8\") pod \"maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p\" (UID: \"48841348-a856-4dcc-9687-b987faf399cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p" Oct 30 19:15:00.312840 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.312729 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21614359-ee0c-4cde-b35f-4638cf40c746-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh\" (UID: \"21614359-ee0c-4cde-b35f-4638cf40c746\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh" Oct 30 19:15:00.312914 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.312847 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqrpp\" (UniqueName: \"kubernetes.io/projected/9b3a8087-79cb-4f56-bae0-de62d6c81ca7-kube-api-access-zqrpp\") pod \"maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t\" (UID: \"9b3a8087-79cb-4f56-bae0-de62d6c81ca7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t" Oct 30 19:15:00.312914 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.312884 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x68pv\" (UniqueName: \"kubernetes.io/projected/1dab7bc1-1c50-454b-b955-aa4c6fb824e9-kube-api-access-x68pv\") pod \"maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs\" (UID: \"1dab7bc1-1c50-454b-b955-aa4c6fb824e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs" Oct 30 19:15:00.313007 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.312921 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc9k2\" (UniqueName: \"kubernetes.io/projected/29c08a40-95ad-4783-9cc4-80677e55f751-kube-api-access-mc9k2\") pod \"maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24\" (UID: \"29c08a40-95ad-4783-9cc4-80677e55f751\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24" Oct 30 19:15:00.313007 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.312975 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c08a40-95ad-4783-9cc4-80677e55f751-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24\" (UID: \"29c08a40-95ad-4783-9cc4-80677e55f751\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24" Oct 30 19:15:00.313079 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.313007 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48841348-a856-4dcc-9687-b987faf399cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p\" (UID: \"48841348-a856-4dcc-9687-b987faf399cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p" Oct 30 19:15:00.313079 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.313039 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dab7bc1-1c50-454b-b955-aa4c6fb824e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs\" (UID: \"1dab7bc1-1c50-454b-b955-aa4c6fb824e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs" Oct 30 19:15:00.313184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.313164 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl75m\" (UniqueName: \"kubernetes.io/projected/21614359-ee0c-4cde-b35f-4638cf40c746-kube-api-access-tl75m\") pod \"maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh\" (UID: \"21614359-ee0c-4cde-b35f-4638cf40c746\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh" Oct 30 19:15:00.313251 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.313234 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b3a8087-79cb-4f56-bae0-de62d6c81ca7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t\" (UID: \"9b3a8087-79cb-4f56-bae0-de62d6c81ca7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t" Oct 30 19:15:00.342198 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.342168 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n"] Oct 30 19:15:00.342571 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.342549 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x"] Oct 30 19:15:00.347560 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.347536 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2"] Oct 30 19:15:00.372893 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.372870 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh"] Oct 30 19:15:00.414096 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.414034 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldg42\" (UniqueName: \"kubernetes.io/projected/ff579236-d055-4e67-864a-40233611afe8-kube-api-access-ldg42\") pod \"maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x\" (UID: \"ff579236-d055-4e67-864a-40233611afe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x" Oct 30 19:15:00.414096 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.414068 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff579236-d055-4e67-864a-40233611afe8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x\" (UID: \"ff579236-d055-4e67-864a-40233611afe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x" Oct 30 19:15:00.414416 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.414099 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c08a40-95ad-4783-9cc4-80677e55f751-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24\" (UID: \"29c08a40-95ad-4783-9cc4-80677e55f751\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24" Oct 30 19:15:00.414416 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.414206 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48841348-a856-4dcc-9687-b987faf399cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p\" (UID: \"48841348-a856-4dcc-9687-b987faf399cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p" Oct 30 19:15:00.414416 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.414243 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dab7bc1-1c50-454b-b955-aa4c6fb824e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs\" (UID: \"1dab7bc1-1c50-454b-b955-aa4c6fb824e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs" Oct 30 19:15:00.414416 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.414265 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46451788-1ba8-4705-a579-9ab31c9a10b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2\" (UID: \"46451788-1ba8-4705-a579-9ab31c9a10b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2" Oct 30 19:15:00.414416 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.414321 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl75m\" (UniqueName: \"kubernetes.io/projected/21614359-ee0c-4cde-b35f-4638cf40c746-kube-api-access-tl75m\") pod \"maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh\" (UID: \"21614359-ee0c-4cde-b35f-4638cf40c746\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh" Oct 30 19:15:00.414416 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.414344 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b3a8087-79cb-4f56-bae0-de62d6c81ca7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t\" (UID: \"9b3a8087-79cb-4f56-bae0-de62d6c81ca7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t" Oct 30 19:15:00.414756 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.414429 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrqb8\" (UniqueName: \"kubernetes.io/projected/48841348-a856-4dcc-9687-b987faf399cd-kube-api-access-wrqb8\") pod \"maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p\" (UID: \"48841348-a856-4dcc-9687-b987faf399cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p" Oct 30 19:15:00.414756 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.414485 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21614359-ee0c-4cde-b35f-4638cf40c746-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh\" (UID: \"21614359-ee0c-4cde-b35f-4638cf40c746\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh" Oct 30 19:15:00.414756 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.414594 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n\" (UID: \"8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n" Oct 30 19:15:00.414756 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.414630 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2xbx\" (UniqueName: \"kubernetes.io/projected/46451788-1ba8-4705-a579-9ab31c9a10b8-kube-api-access-b2xbx\") pod \"maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2\" (UID: \"46451788-1ba8-4705-a579-9ab31c9a10b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2" Oct 30 19:15:00.414756 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.414661 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqrpp\" (UniqueName: \"kubernetes.io/projected/9b3a8087-79cb-4f56-bae0-de62d6c81ca7-kube-api-access-zqrpp\") pod \"maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t\" (UID: \"9b3a8087-79cb-4f56-bae0-de62d6c81ca7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t" Oct 30 19:15:00.414756 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.414693 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x68pv\" (UniqueName: \"kubernetes.io/projected/1dab7bc1-1c50-454b-b955-aa4c6fb824e9-kube-api-access-x68pv\") pod \"maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs\" (UID: \"1dab7bc1-1c50-454b-b955-aa4c6fb824e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs" Oct 30 19:15:00.414756 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.414723 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc9k2\" (UniqueName: \"kubernetes.io/projected/29c08a40-95ad-4783-9cc4-80677e55f751-kube-api-access-mc9k2\") pod \"maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24\" (UID: \"29c08a40-95ad-4783-9cc4-80677e55f751\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24" Oct 30 19:15:00.415192 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.414769 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvt66\" (UniqueName: \"kubernetes.io/projected/8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1-kube-api-access-dvt66\") pod \"maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n\" (UID: \"8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n" Oct 30 19:15:00.415192 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.414995 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dab7bc1-1c50-454b-b955-aa4c6fb824e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs\" (UID: \"1dab7bc1-1c50-454b-b955-aa4c6fb824e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs" Oct 30 19:15:00.415192 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.415011 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c08a40-95ad-4783-9cc4-80677e55f751-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24\" (UID: \"29c08a40-95ad-4783-9cc4-80677e55f751\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24" Oct 30 19:15:00.415192 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.414995 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48841348-a856-4dcc-9687-b987faf399cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p\" (UID: \"48841348-a856-4dcc-9687-b987faf399cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p" Oct 30 19:15:00.415399 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.415247 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b3a8087-79cb-4f56-bae0-de62d6c81ca7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t\" (UID: \"9b3a8087-79cb-4f56-bae0-de62d6c81ca7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t" Oct 30 19:15:00.415549 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.415531 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21614359-ee0c-4cde-b35f-4638cf40c746-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh\" (UID: \"21614359-ee0c-4cde-b35f-4638cf40c746\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh" Oct 30 19:15:00.476094 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.476072 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqrpp\" (UniqueName: \"kubernetes.io/projected/9b3a8087-79cb-4f56-bae0-de62d6c81ca7-kube-api-access-zqrpp\") pod \"maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t\" (UID: \"9b3a8087-79cb-4f56-bae0-de62d6c81ca7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t" Oct 30 19:15:00.476252 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.476231 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl75m\" (UniqueName: \"kubernetes.io/projected/21614359-ee0c-4cde-b35f-4638cf40c746-kube-api-access-tl75m\") pod \"maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh\" (UID: \"21614359-ee0c-4cde-b35f-4638cf40c746\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh" Oct 30 19:15:00.476303 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.476239 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrqb8\" (UniqueName: \"kubernetes.io/projected/48841348-a856-4dcc-9687-b987faf399cd-kube-api-access-wrqb8\") pod \"maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p\" (UID: \"48841348-a856-4dcc-9687-b987faf399cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p" Oct 30 19:15:00.476553 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.476529 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x68pv\" (UniqueName: \"kubernetes.io/projected/1dab7bc1-1c50-454b-b955-aa4c6fb824e9-kube-api-access-x68pv\") pod \"maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs\" (UID: \"1dab7bc1-1c50-454b-b955-aa4c6fb824e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs" Oct 30 19:15:00.476688 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.476636 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc9k2\" (UniqueName: \"kubernetes.io/projected/29c08a40-95ad-4783-9cc4-80677e55f751-kube-api-access-mc9k2\") pod \"maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24\" (UID: \"29c08a40-95ad-4783-9cc4-80677e55f751\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24" Oct 30 19:15:00.480413 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.480391 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs" Oct 30 19:15:00.490096 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.490076 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t" Oct 30 19:15:00.496076 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.496053 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24" Oct 30 19:15:00.518502 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.518472 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2xbx\" (UniqueName: \"kubernetes.io/projected/46451788-1ba8-4705-a579-9ab31c9a10b8-kube-api-access-b2xbx\") pod \"maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2\" (UID: \"46451788-1ba8-4705-a579-9ab31c9a10b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2" Oct 30 19:15:00.518749 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.518703 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvt66\" (UniqueName: \"kubernetes.io/projected/8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1-kube-api-access-dvt66\") pod \"maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n\" (UID: \"8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n" Oct 30 19:15:00.518911 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.518895 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldg42\" (UniqueName: \"kubernetes.io/projected/ff579236-d055-4e67-864a-40233611afe8-kube-api-access-ldg42\") pod \"maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x\" (UID: \"ff579236-d055-4e67-864a-40233611afe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x" Oct 30 19:15:00.519061 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.519045 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff579236-d055-4e67-864a-40233611afe8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x\" (UID: \"ff579236-d055-4e67-864a-40233611afe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x" Oct 30 19:15:00.519264 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.519234 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46451788-1ba8-4705-a579-9ab31c9a10b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2\" (UID: \"46451788-1ba8-4705-a579-9ab31c9a10b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2" Oct 30 19:15:00.519630 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.519611 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n\" (UID: \"8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n" Oct 30 19:15:00.519816 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.519715 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff579236-d055-4e67-864a-40233611afe8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x\" (UID: \"ff579236-d055-4e67-864a-40233611afe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x" Oct 30 19:15:00.519997 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.519974 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46451788-1ba8-4705-a579-9ab31c9a10b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2\" (UID: \"46451788-1ba8-4705-a579-9ab31c9a10b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2" Oct 30 19:15:00.520281 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.520262 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n\" (UID: \"8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n" Oct 30 19:15:00.528785 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.528757 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p" Oct 30 19:15:00.537096 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.537067 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh" Oct 30 19:15:00.585705 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.585594 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvt66\" (UniqueName: \"kubernetes.io/projected/8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1-kube-api-access-dvt66\") pod \"maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n\" (UID: \"8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n" Oct 30 19:15:00.586646 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.586445 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldg42\" (UniqueName: \"kubernetes.io/projected/ff579236-d055-4e67-864a-40233611afe8-kube-api-access-ldg42\") pod \"maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x\" (UID: \"ff579236-d055-4e67-864a-40233611afe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x" Oct 30 19:15:00.591835 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.591814 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2xbx\" (UniqueName: \"kubernetes.io/projected/46451788-1ba8-4705-a579-9ab31c9a10b8-kube-api-access-b2xbx\") pod \"maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2\" (UID: \"46451788-1ba8-4705-a579-9ab31c9a10b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2" Oct 30 19:15:00.843649 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.843621 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2" Oct 30 19:15:00.850021 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.849995 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x" Oct 30 19:15:00.875001 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.874962 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n" Oct 30 19:15:00.889780 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.889748 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs"] Oct 30 19:15:00.893099 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:15:00.893066 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dab7bc1_1c50_454b_b955_aa4c6fb824e9.slice/crio-505ab5ceeb2fabefe3d9f21988a77f6da70c89f8770d4d5c876bf9446676cc48 WatchSource:0}: Error finding container 505ab5ceeb2fabefe3d9f21988a77f6da70c89f8770d4d5c876bf9446676cc48: Status 404 returned error can't find the container with id 505ab5ceeb2fabefe3d9f21988a77f6da70c89f8770d4d5c876bf9446676cc48 Oct 30 19:15:00.928701 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.928672 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t"] Oct 30 19:15:00.999329 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:00.999079 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p"] Oct 30 19:15:01.005027 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:01.003835 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh"] Oct 30 19:15:01.008884 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:01.008782 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24"] Oct 30 19:15:01.181584 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:01.181525 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh" event={"ID":"21614359-ee0c-4cde-b35f-4638cf40c746","Type":"ContainerStarted","Data":"a925fa745617c67c45db8ca2e039fe073a37363736e5f771152f21ac48dd9fb2"} Oct 30 19:15:01.182815 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:01.182793 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t" event={"ID":"9b3a8087-79cb-4f56-bae0-de62d6c81ca7","Type":"ContainerStarted","Data":"4a7bb1e3545f093575eede070d76520df7d19e55d31c9c3036b1a9a73171b85d"} Oct 30 19:15:01.184106 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:01.184081 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p" event={"ID":"48841348-a856-4dcc-9687-b987faf399cd","Type":"ContainerStarted","Data":"71e8b79fd64ee77b33c74fe58183f3d1e229d286405559844758e3844ae4d895"} Oct 30 19:15:01.185265 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:01.185243 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24" event={"ID":"29c08a40-95ad-4783-9cc4-80677e55f751","Type":"ContainerStarted","Data":"7c929f0adfd0227f866aa78064896a990eb0426b9fb43279d9803e520e192e6c"} Oct 30 19:15:01.186418 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:01.186397 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs" event={"ID":"1dab7bc1-1c50-454b-b955-aa4c6fb824e9","Type":"ContainerStarted","Data":"505ab5ceeb2fabefe3d9f21988a77f6da70c89f8770d4d5c876bf9446676cc48"} Oct 30 19:15:01.224190 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:01.224160 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2"] Oct 30 19:15:01.249521 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:01.249497 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x"] Oct 30 19:15:01.290383 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:01.290362 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n"] Oct 30 19:15:01.327869 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:15:01.327832 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46451788_1ba8_4705_a579_9ab31c9a10b8.slice/crio-070968f636a793355d7de97ce9d98a9e0ca9e8a6737b62239a5d9ab492b4df83 WatchSource:0}: Error finding container 070968f636a793355d7de97ce9d98a9e0ca9e8a6737b62239a5d9ab492b4df83: Status 404 returned error can't find the container with id 070968f636a793355d7de97ce9d98a9e0ca9e8a6737b62239a5d9ab492b4df83 Oct 30 19:15:01.855539 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:01.855506 4531 scope.go:122] "RemoveContainer" containerID="186f0e704a2215a78cb0d43ab8f6120cd722fa8183e5305fd1441d4457d4ca7b" Oct 30 19:15:02.195343 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:02.195305 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24" event={"ID":"29c08a40-95ad-4783-9cc4-80677e55f751","Type":"ContainerStarted","Data":"6ed81c3db6f4b1d25ac9fe1affe051ff89f4300566d4a271fa39f6cb5a0bb5b7"} Oct 30 19:15:02.197365 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:02.197333 4531 generic.go:356] "Generic (PLEG): container finished" podID="1dab7bc1-1c50-454b-b955-aa4c6fb824e9" containerID="f2a53676a0ca18b65d8bb9c0569eddb46f893a3b10354e544c016c43328ead1d" exitCode=60 Oct 30 19:15:02.197627 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:02.197430 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs" event={"ID":"1dab7bc1-1c50-454b-b955-aa4c6fb824e9","Type":"ContainerDied","Data":"f2a53676a0ca18b65d8bb9c0569eddb46f893a3b10354e544c016c43328ead1d"} Oct 30 19:15:02.197712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:02.197701 4531 scope.go:122] "RemoveContainer" containerID="f2a53676a0ca18b65d8bb9c0569eddb46f893a3b10354e544c016c43328ead1d" Oct 30 19:15:02.199632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:02.199600 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n" event={"ID":"8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1","Type":"ContainerStarted","Data":"1ad9f2a0508a412c044b4dc5ae9ae26303ec2fbd44d2e14a627bd8448bd713a8"} Oct 30 19:15:02.199745 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:02.199669 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n" event={"ID":"8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1","Type":"ContainerStarted","Data":"d9ec2a4907c3f39a9c6438262db09997debafd5286aa585fffe81d24cb7f3b5a"} Oct 30 19:15:02.203172 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:02.203102 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x" event={"ID":"ff579236-d055-4e67-864a-40233611afe8","Type":"ContainerStarted","Data":"9780b0318bc92f3746a5ff8fdc30bfd3ceff56482052b4974235b58519dc669d"} Oct 30 19:15:02.203395 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:02.203186 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x" event={"ID":"ff579236-d055-4e67-864a-40233611afe8","Type":"ContainerStarted","Data":"f6f81e94eeee48c6527ff6c01449b56dfb3899ee9d519581fc3e04696601c4ac"} Oct 30 19:15:02.205170 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:02.205084 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh" event={"ID":"21614359-ee0c-4cde-b35f-4638cf40c746","Type":"ContainerStarted","Data":"f95a2351d511a2afe09cca00d1d55e09c3762fa2e294d48d7c38ebd0c2b99669"} Oct 30 19:15:02.206904 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:02.206866 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2" event={"ID":"46451788-1ba8-4705-a579-9ab31c9a10b8","Type":"ContainerStarted","Data":"1ca573ea33bceac53f261d3ee06adf33662a11a70be1b5f251029c8f0e050d88"} Oct 30 19:15:02.207084 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:02.206925 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2" event={"ID":"46451788-1ba8-4705-a579-9ab31c9a10b8","Type":"ContainerStarted","Data":"070968f636a793355d7de97ce9d98a9e0ca9e8a6737b62239a5d9ab492b4df83"} Oct 30 19:15:02.208936 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:02.208905 4531 generic.go:356] "Generic (PLEG): container finished" podID="9b3a8087-79cb-4f56-bae0-de62d6c81ca7" containerID="e63de1ab307767a36d5c37cbc5072a58b525e78150d38b58958ae59b36add97e" exitCode=60 Oct 30 19:15:02.209099 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:02.208981 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t" event={"ID":"9b3a8087-79cb-4f56-bae0-de62d6c81ca7","Type":"ContainerDied","Data":"e63de1ab307767a36d5c37cbc5072a58b525e78150d38b58958ae59b36add97e"} Oct 30 19:15:02.209313 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:02.209299 4531 scope.go:122] "RemoveContainer" containerID="e63de1ab307767a36d5c37cbc5072a58b525e78150d38b58958ae59b36add97e" Oct 30 19:15:02.210845 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:02.210816 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p" event={"ID":"48841348-a856-4dcc-9687-b987faf399cd","Type":"ContainerStarted","Data":"f9690c4da8e3db67b57caceb59fde0f3cd61c357b59990196dde01d4b63a4d6b"} Oct 30 19:15:02.321419 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:02.321332 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24" podStartSLOduration=2.321307034 podStartE2EDuration="2.321307034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.31960983 +0000 UTC m=+2462.346714792" watchObservedRunningTime="2026-10-30 19:15:02.321307034 +0000 UTC m=+2462.348411996" Oct 30 19:15:02.700964 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:02.700871 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n" podStartSLOduration=2.700845824 podStartE2EDuration="2.700845824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.697952573 +0000 UTC m=+2462.725057535" watchObservedRunningTime="2026-10-30 19:15:02.700845824 +0000 UTC m=+2462.727950787" Oct 30 19:15:02.826307 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:02.826246 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p" podStartSLOduration=2.826225796 podStartE2EDuration="2.826225796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.825295718 +0000 UTC m=+2462.852400678" watchObservedRunningTime="2026-10-30 19:15:02.826225796 +0000 UTC m=+2462.853330757" Oct 30 19:15:03.000791 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.000731 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2" podStartSLOduration=3.000712448 podStartE2EDuration="3.000712448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.998243142 +0000 UTC m=+2463.025348104" watchObservedRunningTime="2026-10-30 19:15:03.000712448 +0000 UTC m=+2463.027817408" Oct 30 19:15:03.120620 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.120565 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x" podStartSLOduration=3.120551139 podStartE2EDuration="3.120551139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:03.119148449 +0000 UTC m=+2463.146253405" watchObservedRunningTime="2026-10-30 19:15:03.120551139 +0000 UTC m=+2463.147656099" Oct 30 19:15:03.219371 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.219336 4531 generic.go:356] "Generic (PLEG): container finished" podID="29c08a40-95ad-4783-9cc4-80677e55f751" containerID="6ed81c3db6f4b1d25ac9fe1affe051ff89f4300566d4a271fa39f6cb5a0bb5b7" exitCode=60 Oct 30 19:15:03.219820 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.219424 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24" event={"ID":"29c08a40-95ad-4783-9cc4-80677e55f751","Type":"ContainerDied","Data":"6ed81c3db6f4b1d25ac9fe1affe051ff89f4300566d4a271fa39f6cb5a0bb5b7"} Oct 30 19:15:03.219820 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.219649 4531 scope.go:122] "RemoveContainer" containerID="6ed81c3db6f4b1d25ac9fe1affe051ff89f4300566d4a271fa39f6cb5a0bb5b7" Oct 30 19:15:03.221482 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.221456 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs" event={"ID":"1dab7bc1-1c50-454b-b955-aa4c6fb824e9","Type":"ContainerStarted","Data":"d2c1f7506822a05640dda37d1bf29e153214c5df7b90421f25f20b68c005bb55"} Oct 30 19:15:03.222994 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.222972 4531 generic.go:356] "Generic (PLEG): container finished" podID="8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1" containerID="1ad9f2a0508a412c044b4dc5ae9ae26303ec2fbd44d2e14a627bd8448bd713a8" exitCode=60 Oct 30 19:15:03.223047 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.223021 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n" event={"ID":"8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1","Type":"ContainerDied","Data":"1ad9f2a0508a412c044b4dc5ae9ae26303ec2fbd44d2e14a627bd8448bd713a8"} Oct 30 19:15:03.223269 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.223254 4531 scope.go:122] "RemoveContainer" containerID="1ad9f2a0508a412c044b4dc5ae9ae26303ec2fbd44d2e14a627bd8448bd713a8" Oct 30 19:15:03.224788 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.224763 4531 generic.go:356] "Generic (PLEG): container finished" podID="ff579236-d055-4e67-864a-40233611afe8" containerID="9780b0318bc92f3746a5ff8fdc30bfd3ceff56482052b4974235b58519dc669d" exitCode=60 Oct 30 19:15:03.224890 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.224846 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x" event={"ID":"ff579236-d055-4e67-864a-40233611afe8","Type":"ContainerDied","Data":"9780b0318bc92f3746a5ff8fdc30bfd3ceff56482052b4974235b58519dc669d"} Oct 30 19:15:03.225007 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.224994 4531 scope.go:122] "RemoveContainer" containerID="9780b0318bc92f3746a5ff8fdc30bfd3ceff56482052b4974235b58519dc669d" Oct 30 19:15:03.226427 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.226407 4531 generic.go:356] "Generic (PLEG): container finished" podID="21614359-ee0c-4cde-b35f-4638cf40c746" containerID="f95a2351d511a2afe09cca00d1d55e09c3762fa2e294d48d7c38ebd0c2b99669" exitCode=60 Oct 30 19:15:03.226529 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.226481 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh" event={"ID":"21614359-ee0c-4cde-b35f-4638cf40c746","Type":"ContainerDied","Data":"f95a2351d511a2afe09cca00d1d55e09c3762fa2e294d48d7c38ebd0c2b99669"} Oct 30 19:15:03.226659 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.226644 4531 scope.go:122] "RemoveContainer" containerID="f95a2351d511a2afe09cca00d1d55e09c3762fa2e294d48d7c38ebd0c2b99669" Oct 30 19:15:03.228077 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.228057 4531 generic.go:356] "Generic (PLEG): container finished" podID="46451788-1ba8-4705-a579-9ab31c9a10b8" containerID="1ca573ea33bceac53f261d3ee06adf33662a11a70be1b5f251029c8f0e050d88" exitCode=60 Oct 30 19:15:03.228209 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.228123 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2" event={"ID":"46451788-1ba8-4705-a579-9ab31c9a10b8","Type":"ContainerDied","Data":"1ca573ea33bceac53f261d3ee06adf33662a11a70be1b5f251029c8f0e050d88"} Oct 30 19:15:03.228340 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.228327 4531 scope.go:122] "RemoveContainer" containerID="1ca573ea33bceac53f261d3ee06adf33662a11a70be1b5f251029c8f0e050d88" Oct 30 19:15:03.231865 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.231754 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t" event={"ID":"9b3a8087-79cb-4f56-bae0-de62d6c81ca7","Type":"ContainerStarted","Data":"16cb3267b1ea04bba99d90e73feb9fc4e76af2320762fe9c62a2d782371393ba"} Oct 30 19:15:03.234738 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.234713 4531 generic.go:356] "Generic (PLEG): container finished" podID="48841348-a856-4dcc-9687-b987faf399cd" containerID="f9690c4da8e3db67b57caceb59fde0f3cd61c357b59990196dde01d4b63a4d6b" exitCode=60 Oct 30 19:15:03.234960 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.234753 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p" event={"ID":"48841348-a856-4dcc-9687-b987faf399cd","Type":"ContainerDied","Data":"f9690c4da8e3db67b57caceb59fde0f3cd61c357b59990196dde01d4b63a4d6b"} Oct 30 19:15:03.235444 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.235017 4531 scope.go:122] "RemoveContainer" containerID="f9690c4da8e3db67b57caceb59fde0f3cd61c357b59990196dde01d4b63a4d6b" Oct 30 19:15:03.237585 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.237551 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh" podStartSLOduration=3.237540067 podStartE2EDuration="3.237540067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:03.234419149 +0000 UTC m=+2463.261524112" watchObservedRunningTime="2026-10-30 19:15:03.237540067 +0000 UTC m=+2463.264645034" Oct 30 19:15:03.638222 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.638193 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h"] Oct 30 19:15:03.642201 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.642121 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h" Oct 30 19:15:03.659613 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.659591 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr44w\" (UniqueName: \"kubernetes.io/projected/4cefcca3-b31c-4b38-9995-650aa594f2cd-kube-api-access-tr44w\") pod \"payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h\" (UID: \"4cefcca3-b31c-4b38-9995-650aa594f2cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h" Oct 30 19:15:03.659731 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.659661 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cefcca3-b31c-4b38-9995-650aa594f2cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h\" (UID: \"4cefcca3-b31c-4b38-9995-650aa594f2cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h" Oct 30 19:15:03.694627 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.694601 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-52d383-dockercfg-hs478\"" Oct 30 19:15:03.694732 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.694641 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-52d383\"" Oct 30 19:15:03.703986 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.703963 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h"] Oct 30 19:15:03.760563 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.760524 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr44w\" (UniqueName: \"kubernetes.io/projected/4cefcca3-b31c-4b38-9995-650aa594f2cd-kube-api-access-tr44w\") pod \"payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h\" (UID: \"4cefcca3-b31c-4b38-9995-650aa594f2cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h" Oct 30 19:15:03.760719 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.760611 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cefcca3-b31c-4b38-9995-650aa594f2cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h\" (UID: \"4cefcca3-b31c-4b38-9995-650aa594f2cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h" Oct 30 19:15:03.761337 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.761318 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cefcca3-b31c-4b38-9995-650aa594f2cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h\" (UID: \"4cefcca3-b31c-4b38-9995-650aa594f2cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h" Oct 30 19:15:03.822392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.822324 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr44w\" (UniqueName: \"kubernetes.io/projected/4cefcca3-b31c-4b38-9995-650aa594f2cd-kube-api-access-tr44w\") pod \"payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h\" (UID: \"4cefcca3-b31c-4b38-9995-650aa594f2cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h" Oct 30 19:15:03.824430 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.824387 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs" podStartSLOduration=3.824374568 podStartE2EDuration="3.824374568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:03.821628861 +0000 UTC m=+2463.848733821" watchObservedRunningTime="2026-10-30 19:15:03.824374568 +0000 UTC m=+2463.851479547" Oct 30 19:15:03.937831 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.937496 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t" podStartSLOduration=3.937477045 podStartE2EDuration="3.937477045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:03.936961094 +0000 UTC m=+2463.964066054" watchObservedRunningTime="2026-10-30 19:15:03.937477045 +0000 UTC m=+2463.964582006" Oct 30 19:15:03.956437 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:03.956215 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h" Oct 30 19:15:04.246317 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.246279 4531 generic.go:356] "Generic (PLEG): container finished" podID="8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1" containerID="9fc4fa02cd1b3de3889892654d3f99d25fe8b1416e675edf18df4a1c8f35c799" exitCode=60 Oct 30 19:15:04.246644 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.246343 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n" event={"ID":"8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1","Type":"ContainerDied","Data":"9fc4fa02cd1b3de3889892654d3f99d25fe8b1416e675edf18df4a1c8f35c799"} Oct 30 19:15:04.246644 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.246403 4531 scope.go:122] "RemoveContainer" containerID="1ad9f2a0508a412c044b4dc5ae9ae26303ec2fbd44d2e14a627bd8448bd713a8" Oct 30 19:15:04.246749 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.246650 4531 scope.go:122] "RemoveContainer" containerID="9fc4fa02cd1b3de3889892654d3f99d25fe8b1416e675edf18df4a1c8f35c799" Oct 30 19:15:04.246976 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:04.246949 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n_odh-ai-gateway-infra(8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n" podUID="8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1" Oct 30 19:15:04.248322 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.248298 4531 generic.go:356] "Generic (PLEG): container finished" podID="ff579236-d055-4e67-864a-40233611afe8" containerID="29a83d84912cfd5a0fd8535074a4b3da27695dbd5b17f068aa8ae8fdefba1ee2" exitCode=60 Oct 30 19:15:04.248422 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.248385 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x" event={"ID":"ff579236-d055-4e67-864a-40233611afe8","Type":"ContainerDied","Data":"29a83d84912cfd5a0fd8535074a4b3da27695dbd5b17f068aa8ae8fdefba1ee2"} Oct 30 19:15:04.248661 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.248648 4531 scope.go:122] "RemoveContainer" containerID="29a83d84912cfd5a0fd8535074a4b3da27695dbd5b17f068aa8ae8fdefba1ee2" Oct 30 19:15:04.248917 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:04.248892 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x_odh-ai-gateway-infra(ff579236-d055-4e67-864a-40233611afe8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x" podUID="ff579236-d055-4e67-864a-40233611afe8" Oct 30 19:15:04.250207 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.250181 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh" event={"ID":"21614359-ee0c-4cde-b35f-4638cf40c746","Type":"ContainerStarted","Data":"bfb55d47f54e0170a2d0d6a5556ac15b7b14b6c39cb5905e493db911f153f5aa"} Oct 30 19:15:04.252372 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.252350 4531 generic.go:356] "Generic (PLEG): container finished" podID="46451788-1ba8-4705-a579-9ab31c9a10b8" containerID="f11418e31c384161933c41181efbc2361b80ee828b6476b6f71dacd73f61a8f1" exitCode=60 Oct 30 19:15:04.252470 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.252400 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2" event={"ID":"46451788-1ba8-4705-a579-9ab31c9a10b8","Type":"ContainerDied","Data":"f11418e31c384161933c41181efbc2361b80ee828b6476b6f71dacd73f61a8f1"} Oct 30 19:15:04.252673 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.252654 4531 scope.go:122] "RemoveContainer" containerID="f11418e31c384161933c41181efbc2361b80ee828b6476b6f71dacd73f61a8f1" Oct 30 19:15:04.252923 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:04.252904 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2_odh-ai-gateway-infra(46451788-1ba8-4705-a579-9ab31c9a10b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2" podUID="46451788-1ba8-4705-a579-9ab31c9a10b8" Oct 30 19:15:04.254368 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.254349 4531 generic.go:356] "Generic (PLEG): container finished" podID="9b3a8087-79cb-4f56-bae0-de62d6c81ca7" containerID="16cb3267b1ea04bba99d90e73feb9fc4e76af2320762fe9c62a2d782371393ba" exitCode=60 Oct 30 19:15:04.254462 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.254409 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t" event={"ID":"9b3a8087-79cb-4f56-bae0-de62d6c81ca7","Type":"ContainerDied","Data":"16cb3267b1ea04bba99d90e73feb9fc4e76af2320762fe9c62a2d782371393ba"} Oct 30 19:15:04.254655 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.254641 4531 scope.go:122] "RemoveContainer" containerID="16cb3267b1ea04bba99d90e73feb9fc4e76af2320762fe9c62a2d782371393ba" Oct 30 19:15:04.254869 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:04.254841 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t_odh-ai-gateway-infra(9b3a8087-79cb-4f56-bae0-de62d6c81ca7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t" podUID="9b3a8087-79cb-4f56-bae0-de62d6c81ca7" Oct 30 19:15:04.256412 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.256392 4531 generic.go:356] "Generic (PLEG): container finished" podID="48841348-a856-4dcc-9687-b987faf399cd" containerID="f7c53514c7bf6e4dd01287fe7fe19bc15d483a8c957a015d55870129c19b7764" exitCode=60 Oct 30 19:15:04.256500 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.256468 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p" event={"ID":"48841348-a856-4dcc-9687-b987faf399cd","Type":"ContainerDied","Data":"f7c53514c7bf6e4dd01287fe7fe19bc15d483a8c957a015d55870129c19b7764"} Oct 30 19:15:04.256723 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.256704 4531 scope.go:122] "RemoveContainer" containerID="f7c53514c7bf6e4dd01287fe7fe19bc15d483a8c957a015d55870129c19b7764" Oct 30 19:15:04.256961 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:04.256937 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p_odh-ai-gateway-infra(48841348-a856-4dcc-9687-b987faf399cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p" podUID="48841348-a856-4dcc-9687-b987faf399cd" Oct 30 19:15:04.258787 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.258410 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24" event={"ID":"29c08a40-95ad-4783-9cc4-80677e55f751","Type":"ContainerStarted","Data":"f44dc06ffd9bbbe724bcbfc8fc3fabd0e01c90f1f688fa4ddcd076ba646f9bc6"} Oct 30 19:15:04.260338 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.260315 4531 generic.go:356] "Generic (PLEG): container finished" podID="1dab7bc1-1c50-454b-b955-aa4c6fb824e9" containerID="d2c1f7506822a05640dda37d1bf29e153214c5df7b90421f25f20b68c005bb55" exitCode=60 Oct 30 19:15:04.260427 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.260371 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs" event={"ID":"1dab7bc1-1c50-454b-b955-aa4c6fb824e9","Type":"ContainerDied","Data":"d2c1f7506822a05640dda37d1bf29e153214c5df7b90421f25f20b68c005bb55"} Oct 30 19:15:04.260593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.260576 4531 scope.go:122] "RemoveContainer" containerID="d2c1f7506822a05640dda37d1bf29e153214c5df7b90421f25f20b68c005bb55" Oct 30 19:15:04.260792 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:04.260775 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs_odh-ai-gateway-infra(1dab7bc1-1c50-454b-b955-aa4c6fb824e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs" podUID="1dab7bc1-1c50-454b-b955-aa4c6fb824e9" Oct 30 19:15:04.457870 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.457846 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h"] Oct 30 19:15:04.483998 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:15:04.483971 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cefcca3_b31c_4b38_9995_650aa594f2cd.slice/crio-1bf3dfb4ebd65902dd77f78cf3940e026b9fa69f6fd678f424636840efdfe784 WatchSource:0}: Error finding container 1bf3dfb4ebd65902dd77f78cf3940e026b9fa69f6fd678f424636840efdfe784: Status 404 returned error can't find the container with id 1bf3dfb4ebd65902dd77f78cf3940e026b9fa69f6fd678f424636840efdfe784 Oct 30 19:15:04.484113 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.484058 4531 scope.go:122] "RemoveContainer" containerID="9780b0318bc92f3746a5ff8fdc30bfd3ceff56482052b4974235b58519dc669d" Oct 30 19:15:04.500263 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.500243 4531 scope.go:122] "RemoveContainer" containerID="1ca573ea33bceac53f261d3ee06adf33662a11a70be1b5f251029c8f0e050d88" Oct 30 19:15:04.520711 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.520690 4531 scope.go:122] "RemoveContainer" containerID="e63de1ab307767a36d5c37cbc5072a58b525e78150d38b58958ae59b36add97e" Oct 30 19:15:04.541179 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.541162 4531 scope.go:122] "RemoveContainer" containerID="f9690c4da8e3db67b57caceb59fde0f3cd61c357b59990196dde01d4b63a4d6b" Oct 30 19:15:04.553487 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:04.553466 4531 scope.go:122] "RemoveContainer" containerID="f2a53676a0ca18b65d8bb9c0569eddb46f893a3b10354e544c016c43328ead1d" Oct 30 19:15:05.268893 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:05.268863 4531 scope.go:122] "RemoveContainer" containerID="f7c53514c7bf6e4dd01287fe7fe19bc15d483a8c957a015d55870129c19b7764" Oct 30 19:15:05.269326 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:05.269231 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p_odh-ai-gateway-infra(48841348-a856-4dcc-9687-b987faf399cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p" podUID="48841348-a856-4dcc-9687-b987faf399cd" Oct 30 19:15:05.270518 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:05.270494 4531 generic.go:356] "Generic (PLEG): container finished" podID="29c08a40-95ad-4783-9cc4-80677e55f751" containerID="f44dc06ffd9bbbe724bcbfc8fc3fabd0e01c90f1f688fa4ddcd076ba646f9bc6" exitCode=60 Oct 30 19:15:05.270645 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:05.270570 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24" event={"ID":"29c08a40-95ad-4783-9cc4-80677e55f751","Type":"ContainerDied","Data":"f44dc06ffd9bbbe724bcbfc8fc3fabd0e01c90f1f688fa4ddcd076ba646f9bc6"} Oct 30 19:15:05.270694 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:05.270656 4531 scope.go:122] "RemoveContainer" containerID="6ed81c3db6f4b1d25ac9fe1affe051ff89f4300566d4a271fa39f6cb5a0bb5b7" Oct 30 19:15:05.270939 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:05.270919 4531 scope.go:122] "RemoveContainer" containerID="f44dc06ffd9bbbe724bcbfc8fc3fabd0e01c90f1f688fa4ddcd076ba646f9bc6" Oct 30 19:15:05.271241 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:05.271215 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24_odh-ai-gateway-infra(29c08a40-95ad-4783-9cc4-80677e55f751)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24" podUID="29c08a40-95ad-4783-9cc4-80677e55f751" Oct 30 19:15:05.272454 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:05.272435 4531 scope.go:122] "RemoveContainer" containerID="d2c1f7506822a05640dda37d1bf29e153214c5df7b90421f25f20b68c005bb55" Oct 30 19:15:05.272708 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:05.272686 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs_odh-ai-gateway-infra(1dab7bc1-1c50-454b-b955-aa4c6fb824e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs" podUID="1dab7bc1-1c50-454b-b955-aa4c6fb824e9" Oct 30 19:15:05.273933 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:05.273913 4531 scope.go:122] "RemoveContainer" containerID="9fc4fa02cd1b3de3889892654d3f99d25fe8b1416e675edf18df4a1c8f35c799" Oct 30 19:15:05.274234 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:05.274206 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n_odh-ai-gateway-infra(8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n" podUID="8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1" Oct 30 19:15:05.275255 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:05.275230 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h" event={"ID":"4cefcca3-b31c-4b38-9995-650aa594f2cd","Type":"ContainerStarted","Data":"7c7e513e31687daa1f37a14a961c745e811cf32a3759d04f9521ba5f76b19b5b"} Oct 30 19:15:05.275347 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:05.275276 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h" event={"ID":"4cefcca3-b31c-4b38-9995-650aa594f2cd","Type":"ContainerStarted","Data":"1bf3dfb4ebd65902dd77f78cf3940e026b9fa69f6fd678f424636840efdfe784"} Oct 30 19:15:05.275347 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:05.275297 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h" Oct 30 19:15:05.277369 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:05.277347 4531 scope.go:122] "RemoveContainer" containerID="29a83d84912cfd5a0fd8535074a4b3da27695dbd5b17f068aa8ae8fdefba1ee2" Oct 30 19:15:05.277609 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:05.277583 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x_odh-ai-gateway-infra(ff579236-d055-4e67-864a-40233611afe8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x" podUID="ff579236-d055-4e67-864a-40233611afe8" Oct 30 19:15:05.279105 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:05.279084 4531 generic.go:356] "Generic (PLEG): container finished" podID="21614359-ee0c-4cde-b35f-4638cf40c746" containerID="bfb55d47f54e0170a2d0d6a5556ac15b7b14b6c39cb5905e493db911f153f5aa" exitCode=60 Oct 30 19:15:05.279211 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:05.279189 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh" event={"ID":"21614359-ee0c-4cde-b35f-4638cf40c746","Type":"ContainerDied","Data":"bfb55d47f54e0170a2d0d6a5556ac15b7b14b6c39cb5905e493db911f153f5aa"} Oct 30 19:15:05.279384 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:05.279367 4531 scope.go:122] "RemoveContainer" containerID="bfb55d47f54e0170a2d0d6a5556ac15b7b14b6c39cb5905e493db911f153f5aa" Oct 30 19:15:05.279594 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:05.279575 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh_odh-ai-gateway-infra(21614359-ee0c-4cde-b35f-4638cf40c746)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh" podUID="21614359-ee0c-4cde-b35f-4638cf40c746" Oct 30 19:15:05.281210 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:05.281193 4531 scope.go:122] "RemoveContainer" containerID="f11418e31c384161933c41181efbc2361b80ee828b6476b6f71dacd73f61a8f1" Oct 30 19:15:05.281412 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:05.281393 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2_odh-ai-gateway-infra(46451788-1ba8-4705-a579-9ab31c9a10b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2" podUID="46451788-1ba8-4705-a579-9ab31c9a10b8" Oct 30 19:15:05.282940 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:05.282920 4531 scope.go:122] "RemoveContainer" containerID="16cb3267b1ea04bba99d90e73feb9fc4e76af2320762fe9c62a2d782371393ba" Oct 30 19:15:05.283206 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:05.283183 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t_odh-ai-gateway-infra(9b3a8087-79cb-4f56-bae0-de62d6c81ca7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t" podUID="9b3a8087-79cb-4f56-bae0-de62d6c81ca7" Oct 30 19:15:05.285654 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:05.285635 4531 scope.go:122] "RemoveContainer" containerID="f95a2351d511a2afe09cca00d1d55e09c3762fa2e294d48d7c38ebd0c2b99669" Oct 30 19:15:06.202089 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:06.202012 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h" podStartSLOduration=3.201995277 podStartE2EDuration="3.201995277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:06.198941423 +0000 UTC m=+2466.226046383" watchObservedRunningTime="2026-10-30 19:15:06.201995277 +0000 UTC m=+2466.229100260" Oct 30 19:15:06.291203 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:06.291178 4531 scope.go:122] "RemoveContainer" containerID="f44dc06ffd9bbbe724bcbfc8fc3fabd0e01c90f1f688fa4ddcd076ba646f9bc6" Oct 30 19:15:06.291687 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:06.291485 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24_odh-ai-gateway-infra(29c08a40-95ad-4783-9cc4-80677e55f751)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24" podUID="29c08a40-95ad-4783-9cc4-80677e55f751" Oct 30 19:15:06.293187 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:06.293121 4531 scope.go:122] "RemoveContainer" containerID="bfb55d47f54e0170a2d0d6a5556ac15b7b14b6c39cb5905e493db911f153f5aa" Oct 30 19:15:06.293482 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:06.293460 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh_odh-ai-gateway-infra(21614359-ee0c-4cde-b35f-4638cf40c746)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh" podUID="21614359-ee0c-4cde-b35f-4638cf40c746" Oct 30 19:15:15.092035 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:15.091951 4531 scope.go:122] "RemoveContainer" containerID="bfb55d47f54e0170a2d0d6a5556ac15b7b14b6c39cb5905e493db911f153f5aa" Oct 30 19:15:15.092035 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:15.092012 4531 scope.go:122] "RemoveContainer" containerID="f7c53514c7bf6e4dd01287fe7fe19bc15d483a8c957a015d55870129c19b7764" Oct 30 19:15:15.990178 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:15.990114 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n"] Oct 30 19:15:16.092257 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:16.092176 4531 scope.go:122] "RemoveContainer" containerID="f11418e31c384161933c41181efbc2361b80ee828b6476b6f71dacd73f61a8f1" Oct 30 19:15:16.201183 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:16.198389 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n" Oct 30 19:15:16.295919 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:16.295883 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h" Oct 30 19:15:16.300748 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:16.300720 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1-maas-api-service-ca\") pod \"8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1\" (UID: \"8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1\") " Oct 30 19:15:16.300862 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:16.300847 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1-kube-api-access-dvt66\" (UniqueName: \"kubernetes.io/projected/8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1-kube-api-access-dvt66\") pod \"8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1\" (UID: \"8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1\") " Oct 30 19:15:16.301205 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:16.301178 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1-maas-api-service-ca" pod "8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1" (UID: "8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:16.316606 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:16.316535 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1-kube-api-access-dvt66" pod "8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1" (UID: "8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1"). InnerVolumeSpecName "kube-api-access-dvt66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:16.370966 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:16.370923 4531 generic.go:356] "Generic (PLEG): container finished" podID="21614359-ee0c-4cde-b35f-4638cf40c746" containerID="51fdb9e0f15c7dc77855f10f8fc115caf4b3f5ddbaf69aa6216eac25ffa5d477" exitCode=60 Oct 30 19:15:16.371210 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:16.371001 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh" event={"ID":"21614359-ee0c-4cde-b35f-4638cf40c746","Type":"ContainerDied","Data":"51fdb9e0f15c7dc77855f10f8fc115caf4b3f5ddbaf69aa6216eac25ffa5d477"} Oct 30 19:15:16.371210 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:16.371080 4531 scope.go:122] "RemoveContainer" containerID="bfb55d47f54e0170a2d0d6a5556ac15b7b14b6c39cb5905e493db911f153f5aa" Oct 30 19:15:16.371581 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:16.371563 4531 scope.go:122] "RemoveContainer" containerID="51fdb9e0f15c7dc77855f10f8fc115caf4b3f5ddbaf69aa6216eac25ffa5d477" Oct 30 19:15:16.371912 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:16.371882 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh_odh-ai-gateway-infra(21614359-ee0c-4cde-b35f-4638cf40c746)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh" podUID="21614359-ee0c-4cde-b35f-4638cf40c746" Oct 30 19:15:16.373555 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:16.373528 4531 generic.go:356] "Generic (PLEG): container finished" podID="48841348-a856-4dcc-9687-b987faf399cd" containerID="f87fd59387bb7b2c2487a222d0ecab47ac1b6f7ce63a9f53d91ebdff19127557" exitCode=60 Oct 30 19:15:16.373684 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:16.373599 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p" event={"ID":"48841348-a856-4dcc-9687-b987faf399cd","Type":"ContainerDied","Data":"f87fd59387bb7b2c2487a222d0ecab47ac1b6f7ce63a9f53d91ebdff19127557"} Oct 30 19:15:16.374075 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:16.374054 4531 scope.go:122] "RemoveContainer" containerID="f87fd59387bb7b2c2487a222d0ecab47ac1b6f7ce63a9f53d91ebdff19127557" Oct 30 19:15:16.374402 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:16.374380 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p_odh-ai-gateway-infra(48841348-a856-4dcc-9687-b987faf399cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p" podUID="48841348-a856-4dcc-9687-b987faf399cd" Oct 30 19:15:16.375187 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:16.375163 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n" event={"ID":"8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1","Type":"ContainerDied","Data":"d9ec2a4907c3f39a9c6438262db09997debafd5286aa585fffe81d24cb7f3b5a"} Oct 30 19:15:16.375295 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:16.375188 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n" Oct 30 19:15:16.388570 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:16.388543 4531 scope.go:122] "RemoveContainer" containerID="f7c53514c7bf6e4dd01287fe7fe19bc15d483a8c957a015d55870129c19b7764" Oct 30 19:15:16.402101 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:16.402067 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.402101 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:16.402096 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvt66\" (UniqueName: \"kubernetes.io/projected/8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1-kube-api-access-dvt66\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:16.403362 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:16.403341 4531 scope.go:122] "RemoveContainer" containerID="9fc4fa02cd1b3de3889892654d3f99d25fe8b1416e675edf18df4a1c8f35c799" Oct 30 19:15:16.660971 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:16.660869 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n"] Oct 30 19:15:16.678082 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:16.678036 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0be4a4-29889795-wcz7n"] Oct 30 19:15:17.092200 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:17.092114 4531 scope.go:122] "RemoveContainer" containerID="d2c1f7506822a05640dda37d1bf29e153214c5df7b90421f25f20b68c005bb55" Oct 30 19:15:17.386868 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:17.386771 4531 generic.go:356] "Generic (PLEG): container finished" podID="46451788-1ba8-4705-a579-9ab31c9a10b8" containerID="d31ae0f28adfc3ea0e2c8e55050f479b0d712934ffd959372168ebf587f49cd6" exitCode=60 Oct 30 19:15:17.386868 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:17.386848 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2" event={"ID":"46451788-1ba8-4705-a579-9ab31c9a10b8","Type":"ContainerDied","Data":"d31ae0f28adfc3ea0e2c8e55050f479b0d712934ffd959372168ebf587f49cd6"} Oct 30 19:15:17.387410 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:17.386974 4531 scope.go:122] "RemoveContainer" containerID="f11418e31c384161933c41181efbc2361b80ee828b6476b6f71dacd73f61a8f1" Oct 30 19:15:17.387476 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:17.387422 4531 scope.go:122] "RemoveContainer" containerID="d31ae0f28adfc3ea0e2c8e55050f479b0d712934ffd959372168ebf587f49cd6" Oct 30 19:15:17.387778 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:17.387743 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2_odh-ai-gateway-infra(46451788-1ba8-4705-a579-9ab31c9a10b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2" podUID="46451788-1ba8-4705-a579-9ab31c9a10b8" Oct 30 19:15:17.560446 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:17.560404 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh"] Oct 30 19:15:17.777658 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:17.777621 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh" Oct 30 19:15:17.804242 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:17.804194 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p"] Oct 30 19:15:17.927077 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:17.926720 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21614359-ee0c-4cde-b35f-4638cf40c746-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21614359-ee0c-4cde-b35f-4638cf40c746-maas-api-service-ca\") pod \"21614359-ee0c-4cde-b35f-4638cf40c746\" (UID: \"21614359-ee0c-4cde-b35f-4638cf40c746\") " Oct 30 19:15:17.927077 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:17.926862 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21614359-ee0c-4cde-b35f-4638cf40c746-kube-api-access-tl75m\" (UniqueName: \"kubernetes.io/projected/21614359-ee0c-4cde-b35f-4638cf40c746-kube-api-access-tl75m\") pod \"21614359-ee0c-4cde-b35f-4638cf40c746\" (UID: \"21614359-ee0c-4cde-b35f-4638cf40c746\") " Oct 30 19:15:17.928445 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:17.928387 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21614359-ee0c-4cde-b35f-4638cf40c746-maas-api-service-ca" pod "21614359-ee0c-4cde-b35f-4638cf40c746" (UID: "21614359-ee0c-4cde-b35f-4638cf40c746"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:17.935473 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:17.935433 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21614359-ee0c-4cde-b35f-4638cf40c746-kube-api-access-tl75m" pod "21614359-ee0c-4cde-b35f-4638cf40c746" (UID: "21614359-ee0c-4cde-b35f-4638cf40c746"). InnerVolumeSpecName "kube-api-access-tl75m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:17.998593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:17.998559 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p" Oct 30 19:15:18.019675 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.019628 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7"] Oct 30 19:15:18.025563 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.025518 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0be4a4-zl8z7"] Oct 30 19:15:18.028528 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.028494 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl75m\" (UniqueName: \"kubernetes.io/projected/21614359-ee0c-4cde-b35f-4638cf40c746-kube-api-access-tl75m\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:18.028528 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.028528 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21614359-ee0c-4cde-b35f-4638cf40c746-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:18.092149 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.092096 4531 scope.go:122] "RemoveContainer" containerID="16cb3267b1ea04bba99d90e73feb9fc4e76af2320762fe9c62a2d782371393ba" Oct 30 19:15:18.092368 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.092192 4531 scope.go:122] "RemoveContainer" containerID="29a83d84912cfd5a0fd8535074a4b3da27695dbd5b17f068aa8ae8fdefba1ee2" Oct 30 19:15:18.099954 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.099917 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1" path="/var/lib/kubelet/pods/8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1/volumes" Oct 30 19:15:18.100431 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.100414 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af10d82c-0a5c-4139-958c-47721326cf4c" path="/var/lib/kubelet/pods/af10d82c-0a5c-4139-958c-47721326cf4c/volumes" Oct 30 19:15:18.129626 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.129586 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48841348-a856-4dcc-9687-b987faf399cd-kube-api-access-wrqb8\" (UniqueName: \"kubernetes.io/projected/48841348-a856-4dcc-9687-b987faf399cd-kube-api-access-wrqb8\") pod \"48841348-a856-4dcc-9687-b987faf399cd\" (UID: \"48841348-a856-4dcc-9687-b987faf399cd\") " Oct 30 19:15:18.129900 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.129881 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48841348-a856-4dcc-9687-b987faf399cd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48841348-a856-4dcc-9687-b987faf399cd-maas-api-service-ca\") pod \"48841348-a856-4dcc-9687-b987faf399cd\" (UID: \"48841348-a856-4dcc-9687-b987faf399cd\") " Oct 30 19:15:18.130255 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.130223 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48841348-a856-4dcc-9687-b987faf399cd-maas-api-service-ca" pod "48841348-a856-4dcc-9687-b987faf399cd" (UID: "48841348-a856-4dcc-9687-b987faf399cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:18.132067 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.132029 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48841348-a856-4dcc-9687-b987faf399cd-kube-api-access-wrqb8" pod "48841348-a856-4dcc-9687-b987faf399cd" (UID: "48841348-a856-4dcc-9687-b987faf399cd"). InnerVolumeSpecName "kube-api-access-wrqb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:18.231489 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.231370 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48841348-a856-4dcc-9687-b987faf399cd-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:18.231489 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.231408 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrqb8\" (UniqueName: \"kubernetes.io/projected/48841348-a856-4dcc-9687-b987faf399cd-kube-api-access-wrqb8\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:18.397315 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.397277 4531 generic.go:356] "Generic (PLEG): container finished" podID="1dab7bc1-1c50-454b-b955-aa4c6fb824e9" containerID="06280f1d190610b74f9e08d4bce4dc22bfc6276b611ba17a78227c2490c6b3ad" exitCode=60 Oct 30 19:15:18.397776 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.397354 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs" event={"ID":"1dab7bc1-1c50-454b-b955-aa4c6fb824e9","Type":"ContainerDied","Data":"06280f1d190610b74f9e08d4bce4dc22bfc6276b611ba17a78227c2490c6b3ad"} Oct 30 19:15:18.397776 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.397431 4531 scope.go:122] "RemoveContainer" containerID="d2c1f7506822a05640dda37d1bf29e153214c5df7b90421f25f20b68c005bb55" Oct 30 19:15:18.397899 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.397843 4531 scope.go:122] "RemoveContainer" containerID="06280f1d190610b74f9e08d4bce4dc22bfc6276b611ba17a78227c2490c6b3ad" Oct 30 19:15:18.398207 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:18.398182 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs_odh-ai-gateway-infra(1dab7bc1-1c50-454b-b955-aa4c6fb824e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs" podUID="1dab7bc1-1c50-454b-b955-aa4c6fb824e9" Oct 30 19:15:18.399750 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.399713 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh" event={"ID":"21614359-ee0c-4cde-b35f-4638cf40c746","Type":"ContainerDied","Data":"a925fa745617c67c45db8ca2e039fe073a37363736e5f771152f21ac48dd9fb2"} Oct 30 19:15:18.399880 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.399724 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh" Oct 30 19:15:18.403423 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.403392 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p" event={"ID":"48841348-a856-4dcc-9687-b987faf399cd","Type":"ContainerDied","Data":"71e8b79fd64ee77b33c74fe58183f3d1e229d286405559844758e3844ae4d895"} Oct 30 19:15:18.403587 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.403436 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p" Oct 30 19:15:18.413220 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.413192 4531 scope.go:122] "RemoveContainer" containerID="51fdb9e0f15c7dc77855f10f8fc115caf4b3f5ddbaf69aa6216eac25ffa5d477" Oct 30 19:15:18.425476 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.425449 4531 scope.go:122] "RemoveContainer" containerID="f87fd59387bb7b2c2487a222d0ecab47ac1b6f7ce63a9f53d91ebdff19127557" Oct 30 19:15:18.525067 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.525029 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2"] Oct 30 19:15:18.628032 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.627991 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh"] Oct 30 19:15:18.634527 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.634487 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cf43e7-29889795-wjrzh"] Oct 30 19:15:18.671402 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.671368 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2" Oct 30 19:15:18.738255 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.738216 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46451788-1ba8-4705-a579-9ab31c9a10b8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46451788-1ba8-4705-a579-9ab31c9a10b8-maas-api-service-ca\") pod \"46451788-1ba8-4705-a579-9ab31c9a10b8\" (UID: \"46451788-1ba8-4705-a579-9ab31c9a10b8\") " Oct 30 19:15:18.738448 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.738273 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46451788-1ba8-4705-a579-9ab31c9a10b8-kube-api-access-b2xbx\" (UniqueName: \"kubernetes.io/projected/46451788-1ba8-4705-a579-9ab31c9a10b8-kube-api-access-b2xbx\") pod \"46451788-1ba8-4705-a579-9ab31c9a10b8\" (UID: \"46451788-1ba8-4705-a579-9ab31c9a10b8\") " Oct 30 19:15:18.738681 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.738656 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46451788-1ba8-4705-a579-9ab31c9a10b8-maas-api-service-ca" pod "46451788-1ba8-4705-a579-9ab31c9a10b8" (UID: "46451788-1ba8-4705-a579-9ab31c9a10b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:18.738873 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.738859 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46451788-1ba8-4705-a579-9ab31c9a10b8-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:18.741084 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.741024 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46451788-1ba8-4705-a579-9ab31c9a10b8-kube-api-access-b2xbx" pod "46451788-1ba8-4705-a579-9ab31c9a10b8" (UID: "46451788-1ba8-4705-a579-9ab31c9a10b8"). InnerVolumeSpecName "kube-api-access-b2xbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:18.743433 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.743401 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p"] Oct 30 19:15:18.752578 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.752533 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0c9a2b-29889795-pq65p"] Oct 30 19:15:18.840550 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:18.840445 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2xbx\" (UniqueName: \"kubernetes.io/projected/46451788-1ba8-4705-a579-9ab31c9a10b8-kube-api-access-b2xbx\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:19.092377 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:19.092279 4531 scope.go:122] "RemoveContainer" containerID="f44dc06ffd9bbbe724bcbfc8fc3fabd0e01c90f1f688fa4ddcd076ba646f9bc6" Oct 30 19:15:19.412697 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:19.412604 4531 generic.go:356] "Generic (PLEG): container finished" podID="ff579236-d055-4e67-864a-40233611afe8" containerID="bd090ea1a6b3504fec40b63e96f22f77d372c7bc14648168b12b8fe2949959a5" exitCode=60 Oct 30 19:15:19.412697 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:19.412670 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x" event={"ID":"ff579236-d055-4e67-864a-40233611afe8","Type":"ContainerDied","Data":"bd090ea1a6b3504fec40b63e96f22f77d372c7bc14648168b12b8fe2949959a5"} Oct 30 19:15:19.413237 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:19.412735 4531 scope.go:122] "RemoveContainer" containerID="29a83d84912cfd5a0fd8535074a4b3da27695dbd5b17f068aa8ae8fdefba1ee2" Oct 30 19:15:19.413237 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:19.413205 4531 scope.go:122] "RemoveContainer" containerID="bd090ea1a6b3504fec40b63e96f22f77d372c7bc14648168b12b8fe2949959a5" Oct 30 19:15:19.413575 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:19.413547 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x_odh-ai-gateway-infra(ff579236-d055-4e67-864a-40233611afe8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x" podUID="ff579236-d055-4e67-864a-40233611afe8" Oct 30 19:15:19.415904 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:19.415880 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2" Oct 30 19:15:19.416073 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:19.415881 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2" event={"ID":"46451788-1ba8-4705-a579-9ab31c9a10b8","Type":"ContainerDied","Data":"070968f636a793355d7de97ce9d98a9e0ca9e8a6737b62239a5d9ab492b4df83"} Oct 30 19:15:19.417982 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:19.417953 4531 generic.go:356] "Generic (PLEG): container finished" podID="9b3a8087-79cb-4f56-bae0-de62d6c81ca7" containerID="d4d00e0a509888a2485159d14fb30655b69b5a38bc89836333d1bdf28f6a9b87" exitCode=60 Oct 30 19:15:19.418218 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:19.418031 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t" event={"ID":"9b3a8087-79cb-4f56-bae0-de62d6c81ca7","Type":"ContainerDied","Data":"d4d00e0a509888a2485159d14fb30655b69b5a38bc89836333d1bdf28f6a9b87"} Oct 30 19:15:19.418573 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:19.418551 4531 scope.go:122] "RemoveContainer" containerID="d4d00e0a509888a2485159d14fb30655b69b5a38bc89836333d1bdf28f6a9b87" Oct 30 19:15:19.418879 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:19.418855 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t_odh-ai-gateway-infra(9b3a8087-79cb-4f56-bae0-de62d6c81ca7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t" podUID="9b3a8087-79cb-4f56-bae0-de62d6c81ca7" Oct 30 19:15:19.428168 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:19.428122 4531 scope.go:122] "RemoveContainer" containerID="d31ae0f28adfc3ea0e2c8e55050f479b0d712934ffd959372168ebf587f49cd6" Oct 30 19:15:19.438323 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:19.438298 4531 scope.go:122] "RemoveContainer" containerID="16cb3267b1ea04bba99d90e73feb9fc4e76af2320762fe9c62a2d782371393ba" Oct 30 19:15:19.526690 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:19.526663 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs"] Oct 30 19:15:19.697677 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:19.697642 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs" Oct 30 19:15:19.853054 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:19.853011 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dab7bc1-1c50-454b-b955-aa4c6fb824e9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dab7bc1-1c50-454b-b955-aa4c6fb824e9-maas-api-service-ca\") pod \"1dab7bc1-1c50-454b-b955-aa4c6fb824e9\" (UID: \"1dab7bc1-1c50-454b-b955-aa4c6fb824e9\") " Oct 30 19:15:19.853299 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:19.853099 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dab7bc1-1c50-454b-b955-aa4c6fb824e9-kube-api-access-x68pv\" (UniqueName: \"kubernetes.io/projected/1dab7bc1-1c50-454b-b955-aa4c6fb824e9-kube-api-access-x68pv\") pod \"1dab7bc1-1c50-454b-b955-aa4c6fb824e9\" (UID: \"1dab7bc1-1c50-454b-b955-aa4c6fb824e9\") " Oct 30 19:15:19.853510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:19.853482 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dab7bc1-1c50-454b-b955-aa4c6fb824e9-maas-api-service-ca" pod "1dab7bc1-1c50-454b-b955-aa4c6fb824e9" (UID: "1dab7bc1-1c50-454b-b955-aa4c6fb824e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:19.853732 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:19.853715 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dab7bc1-1c50-454b-b955-aa4c6fb824e9-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:19.855677 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:19.855640 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dab7bc1-1c50-454b-b955-aa4c6fb824e9-kube-api-access-x68pv" pod "1dab7bc1-1c50-454b-b955-aa4c6fb824e9" (UID: "1dab7bc1-1c50-454b-b955-aa4c6fb824e9"). InnerVolumeSpecName "kube-api-access-x68pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:19.901297 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:19.901105 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2"] Oct 30 19:15:19.904520 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:19.904481 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889795-czvf2"] Oct 30 19:15:19.954936 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:19.954827 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x68pv\" (UniqueName: \"kubernetes.io/projected/1dab7bc1-1c50-454b-b955-aa4c6fb824e9-kube-api-access-x68pv\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:20.107856 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:20.107816 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21614359-ee0c-4cde-b35f-4638cf40c746" path="/var/lib/kubelet/pods/21614359-ee0c-4cde-b35f-4638cf40c746/volumes" Oct 30 19:15:20.108328 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:20.108307 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46451788-1ba8-4705-a579-9ab31c9a10b8" path="/var/lib/kubelet/pods/46451788-1ba8-4705-a579-9ab31c9a10b8/volumes" Oct 30 19:15:20.108750 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:20.108735 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48841348-a856-4dcc-9687-b987faf399cd" path="/var/lib/kubelet/pods/48841348-a856-4dcc-9687-b987faf399cd/volumes" Oct 30 19:15:20.434560 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:20.434518 4531 generic.go:356] "Generic (PLEG): container finished" podID="29c08a40-95ad-4783-9cc4-80677e55f751" containerID="b1d9e841c31af82f74228fb3a414aa6c0bbe1016df02da2fcffd4558bc7dd85e" exitCode=60 Oct 30 19:15:20.435008 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:20.434593 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24" event={"ID":"29c08a40-95ad-4783-9cc4-80677e55f751","Type":"ContainerDied","Data":"b1d9e841c31af82f74228fb3a414aa6c0bbe1016df02da2fcffd4558bc7dd85e"} Oct 30 19:15:20.435008 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:20.434680 4531 scope.go:122] "RemoveContainer" containerID="f44dc06ffd9bbbe724bcbfc8fc3fabd0e01c90f1f688fa4ddcd076ba646f9bc6" Oct 30 19:15:20.435178 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:20.435088 4531 scope.go:122] "RemoveContainer" containerID="b1d9e841c31af82f74228fb3a414aa6c0bbe1016df02da2fcffd4558bc7dd85e" Oct 30 19:15:20.435497 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:20.435460 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24_odh-ai-gateway-infra(29c08a40-95ad-4783-9cc4-80677e55f751)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24" podUID="29c08a40-95ad-4783-9cc4-80677e55f751" Oct 30 19:15:20.436085 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:20.436064 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs" event={"ID":"1dab7bc1-1c50-454b-b955-aa4c6fb824e9","Type":"ContainerDied","Data":"505ab5ceeb2fabefe3d9f21988a77f6da70c89f8770d4d5c876bf9446676cc48"} Oct 30 19:15:20.436209 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:20.436096 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs" Oct 30 19:15:20.447197 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:20.447171 4531 scope.go:122] "RemoveContainer" containerID="06280f1d190610b74f9e08d4bce4dc22bfc6276b611ba17a78227c2490c6b3ad" Oct 30 19:15:20.547233 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:20.547189 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x"] Oct 30 19:15:20.694850 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:20.694767 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x" Oct 30 19:15:20.764076 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:20.764041 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff579236-d055-4e67-864a-40233611afe8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff579236-d055-4e67-864a-40233611afe8-maas-api-service-ca\") pod \"ff579236-d055-4e67-864a-40233611afe8\" (UID: \"ff579236-d055-4e67-864a-40233611afe8\") " Oct 30 19:15:20.764282 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:20.764204 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff579236-d055-4e67-864a-40233611afe8-kube-api-access-ldg42\" (UniqueName: \"kubernetes.io/projected/ff579236-d055-4e67-864a-40233611afe8-kube-api-access-ldg42\") pod \"ff579236-d055-4e67-864a-40233611afe8\" (UID: \"ff579236-d055-4e67-864a-40233611afe8\") " Oct 30 19:15:20.764479 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:20.764455 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff579236-d055-4e67-864a-40233611afe8-maas-api-service-ca" pod "ff579236-d055-4e67-864a-40233611afe8" (UID: "ff579236-d055-4e67-864a-40233611afe8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:20.766559 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:20.766528 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff579236-d055-4e67-864a-40233611afe8-kube-api-access-ldg42" pod "ff579236-d055-4e67-864a-40233611afe8" (UID: "ff579236-d055-4e67-864a-40233611afe8"). InnerVolumeSpecName "kube-api-access-ldg42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:20.777955 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:20.777925 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs"] Oct 30 19:15:20.781488 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:20.781460 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fd61e3-29889795-mzqbs"] Oct 30 19:15:20.799001 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:20.798969 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t"] Oct 30 19:15:20.866765 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:20.866730 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldg42\" (UniqueName: \"kubernetes.io/projected/ff579236-d055-4e67-864a-40233611afe8-kube-api-access-ldg42\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:20.866922 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:20.866770 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff579236-d055-4e67-864a-40233611afe8-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:20.971572 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:20.971545 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t" Oct 30 19:15:21.068918 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:21.068881 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b3a8087-79cb-4f56-bae0-de62d6c81ca7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b3a8087-79cb-4f56-bae0-de62d6c81ca7-maas-api-service-ca\") pod \"9b3a8087-79cb-4f56-bae0-de62d6c81ca7\" (UID: \"9b3a8087-79cb-4f56-bae0-de62d6c81ca7\") " Oct 30 19:15:21.069104 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:21.068955 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b3a8087-79cb-4f56-bae0-de62d6c81ca7-kube-api-access-zqrpp\" (UniqueName: \"kubernetes.io/projected/9b3a8087-79cb-4f56-bae0-de62d6c81ca7-kube-api-access-zqrpp\") pod \"9b3a8087-79cb-4f56-bae0-de62d6c81ca7\" (UID: \"9b3a8087-79cb-4f56-bae0-de62d6c81ca7\") " Oct 30 19:15:21.069300 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:21.069279 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b3a8087-79cb-4f56-bae0-de62d6c81ca7-maas-api-service-ca" pod "9b3a8087-79cb-4f56-bae0-de62d6c81ca7" (UID: "9b3a8087-79cb-4f56-bae0-de62d6c81ca7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:21.071218 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:21.071195 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b3a8087-79cb-4f56-bae0-de62d6c81ca7-kube-api-access-zqrpp" pod "9b3a8087-79cb-4f56-bae0-de62d6c81ca7" (UID: "9b3a8087-79cb-4f56-bae0-de62d6c81ca7"). InnerVolumeSpecName "kube-api-access-zqrpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:21.169956 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:21.169923 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b3a8087-79cb-4f56-bae0-de62d6c81ca7-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:21.169956 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:21.169954 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqrpp\" (UniqueName: \"kubernetes.io/projected/9b3a8087-79cb-4f56-bae0-de62d6c81ca7-kube-api-access-zqrpp\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:21.446231 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:21.446199 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x" Oct 30 19:15:21.446231 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:21.446221 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x" event={"ID":"ff579236-d055-4e67-864a-40233611afe8","Type":"ContainerDied","Data":"f6f81e94eeee48c6527ff6c01449b56dfb3899ee9d519581fc3e04696601c4ac"} Oct 30 19:15:21.446787 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:21.446292 4531 scope.go:122] "RemoveContainer" containerID="bd090ea1a6b3504fec40b63e96f22f77d372c7bc14648168b12b8fe2949959a5" Oct 30 19:15:21.447782 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:21.447761 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t" Oct 30 19:15:21.447909 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:21.447800 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t" event={"ID":"9b3a8087-79cb-4f56-bae0-de62d6c81ca7","Type":"ContainerDied","Data":"4a7bb1e3545f093575eede070d76520df7d19e55d31c9c3036b1a9a73171b85d"} Oct 30 19:15:21.459423 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:21.459401 4531 scope.go:122] "RemoveContainer" containerID="d4d00e0a509888a2485159d14fb30655b69b5a38bc89836333d1bdf28f6a9b87" Oct 30 19:15:21.562866 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:21.562819 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24"] Oct 30 19:15:21.612956 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:21.612923 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t"] Oct 30 19:15:21.617471 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:21.617432 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889795-4d52t"] Oct 30 19:15:21.720902 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:21.720873 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24" Oct 30 19:15:21.741194 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:21.741162 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x"] Oct 30 19:15:21.759339 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:21.759307 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889795-nnt4x"] Oct 30 19:15:21.878247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:21.878216 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29c08a40-95ad-4783-9cc4-80677e55f751-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c08a40-95ad-4783-9cc4-80677e55f751-maas-api-service-ca\") pod \"29c08a40-95ad-4783-9cc4-80677e55f751\" (UID: \"29c08a40-95ad-4783-9cc4-80677e55f751\") " Oct 30 19:15:21.878425 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:21.878382 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29c08a40-95ad-4783-9cc4-80677e55f751-kube-api-access-mc9k2\" (UniqueName: \"kubernetes.io/projected/29c08a40-95ad-4783-9cc4-80677e55f751-kube-api-access-mc9k2\") pod \"29c08a40-95ad-4783-9cc4-80677e55f751\" (UID: \"29c08a40-95ad-4783-9cc4-80677e55f751\") " Oct 30 19:15:21.878632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:21.878605 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c08a40-95ad-4783-9cc4-80677e55f751-maas-api-service-ca" pod "29c08a40-95ad-4783-9cc4-80677e55f751" (UID: "29c08a40-95ad-4783-9cc4-80677e55f751"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:21.880553 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:21.880531 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c08a40-95ad-4783-9cc4-80677e55f751-kube-api-access-mc9k2" pod "29c08a40-95ad-4783-9cc4-80677e55f751" (UID: "29c08a40-95ad-4783-9cc4-80677e55f751"). InnerVolumeSpecName "kube-api-access-mc9k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:21.979434 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:21.979355 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc9k2\" (UniqueName: \"kubernetes.io/projected/29c08a40-95ad-4783-9cc4-80677e55f751-kube-api-access-mc9k2\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:21.979434 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:21.979386 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c08a40-95ad-4783-9cc4-80677e55f751-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:22.098986 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:22.098953 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dab7bc1-1c50-454b-b955-aa4c6fb824e9" path="/var/lib/kubelet/pods/1dab7bc1-1c50-454b-b955-aa4c6fb824e9/volumes" Oct 30 19:15:22.099385 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:22.099371 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b3a8087-79cb-4f56-bae0-de62d6c81ca7" path="/var/lib/kubelet/pods/9b3a8087-79cb-4f56-bae0-de62d6c81ca7/volumes" Oct 30 19:15:22.099720 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:22.099708 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff579236-d055-4e67-864a-40233611afe8" path="/var/lib/kubelet/pods/ff579236-d055-4e67-864a-40233611afe8/volumes" Oct 30 19:15:22.455539 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:22.455508 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24" Oct 30 19:15:22.456022 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:22.455542 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24" event={"ID":"29c08a40-95ad-4783-9cc4-80677e55f751","Type":"ContainerDied","Data":"7c929f0adfd0227f866aa78064896a990eb0426b9fb43279d9803e520e192e6c"} Oct 30 19:15:22.456022 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:22.455610 4531 scope.go:122] "RemoveContainer" containerID="b1d9e841c31af82f74228fb3a414aa6c0bbe1016df02da2fcffd4558bc7dd85e" Oct 30 19:15:22.621165 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:22.621119 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24"] Oct 30 19:15:22.627870 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:22.627831 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-f554bb-29889795-92w24"] Oct 30 19:15:24.099092 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:24.099045 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29c08a40-95ad-4783-9cc4-80677e55f751" path="/var/lib/kubelet/pods/29c08a40-95ad-4783-9cc4-80677e55f751/volumes" Oct 30 19:15:26.292932 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.292887 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s"] Oct 30 19:15:26.293509 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293392 4531 cpu_manager.go:414] "containerMap: removing container" podUID="21614359-ee0c-4cde-b35f-4638cf40c746" containerName="cleanup" Oct 30 19:15:26.293509 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293408 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21614359-ee0c-4cde-b35f-4638cf40c746" containerName="cleanup" Oct 30 19:15:26.293509 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293421 4531 cpu_manager.go:414] "containerMap: removing container" podUID="1dab7bc1-1c50-454b-b955-aa4c6fb824e9" containerName="cleanup" Oct 30 19:15:26.293509 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293426 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dab7bc1-1c50-454b-b955-aa4c6fb824e9" containerName="cleanup" Oct 30 19:15:26.293509 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293436 4531 cpu_manager.go:414] "containerMap: removing container" podUID="ff579236-d055-4e67-864a-40233611afe8" containerName="cleanup" Oct 30 19:15:26.293509 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293443 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff579236-d055-4e67-864a-40233611afe8" containerName="cleanup" Oct 30 19:15:26.293509 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293452 4531 cpu_manager.go:414] "containerMap: removing container" podUID="48841348-a856-4dcc-9687-b987faf399cd" containerName="cleanup" Oct 30 19:15:26.293509 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293457 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48841348-a856-4dcc-9687-b987faf399cd" containerName="cleanup" Oct 30 19:15:26.293509 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293463 4531 cpu_manager.go:414] "containerMap: removing container" podUID="21614359-ee0c-4cde-b35f-4638cf40c746" containerName="cleanup" Oct 30 19:15:26.293509 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293469 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21614359-ee0c-4cde-b35f-4638cf40c746" containerName="cleanup" Oct 30 19:15:26.293509 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293476 4531 cpu_manager.go:414] "containerMap: removing container" podUID="46451788-1ba8-4705-a579-9ab31c9a10b8" containerName="cleanup" Oct 30 19:15:26.293509 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293481 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46451788-1ba8-4705-a579-9ab31c9a10b8" containerName="cleanup" Oct 30 19:15:26.293509 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293488 4531 cpu_manager.go:414] "containerMap: removing container" podUID="ff579236-d055-4e67-864a-40233611afe8" containerName="cleanup" Oct 30 19:15:26.293509 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293493 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff579236-d055-4e67-864a-40233611afe8" containerName="cleanup" Oct 30 19:15:26.293509 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293501 4531 cpu_manager.go:414] "containerMap: removing container" podUID="8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1" containerName="cleanup" Oct 30 19:15:26.293509 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293505 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1" containerName="cleanup" Oct 30 19:15:26.293509 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293513 4531 cpu_manager.go:414] "containerMap: removing container" podUID="46451788-1ba8-4705-a579-9ab31c9a10b8" containerName="cleanup" Oct 30 19:15:26.293509 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293518 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46451788-1ba8-4705-a579-9ab31c9a10b8" containerName="cleanup" Oct 30 19:15:26.293509 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293525 4531 cpu_manager.go:414] "containerMap: removing container" podUID="46451788-1ba8-4705-a579-9ab31c9a10b8" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293530 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46451788-1ba8-4705-a579-9ab31c9a10b8" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293539 4531 cpu_manager.go:414] "containerMap: removing container" podUID="29c08a40-95ad-4783-9cc4-80677e55f751" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293543 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29c08a40-95ad-4783-9cc4-80677e55f751" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293548 4531 cpu_manager.go:414] "containerMap: removing container" podUID="8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293553 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293562 4531 cpu_manager.go:414] "containerMap: removing container" podUID="1dab7bc1-1c50-454b-b955-aa4c6fb824e9" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293567 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dab7bc1-1c50-454b-b955-aa4c6fb824e9" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293574 4531 cpu_manager.go:414] "containerMap: removing container" podUID="9b3a8087-79cb-4f56-bae0-de62d6c81ca7" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293578 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b3a8087-79cb-4f56-bae0-de62d6c81ca7" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293588 4531 cpu_manager.go:414] "containerMap: removing container" podUID="29c08a40-95ad-4783-9cc4-80677e55f751" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293593 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29c08a40-95ad-4783-9cc4-80677e55f751" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293599 4531 cpu_manager.go:414] "containerMap: removing container" podUID="48841348-a856-4dcc-9687-b987faf399cd" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293603 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48841348-a856-4dcc-9687-b987faf399cd" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293609 4531 cpu_manager.go:414] "containerMap: removing container" podUID="9b3a8087-79cb-4f56-bae0-de62d6c81ca7" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293614 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b3a8087-79cb-4f56-bae0-de62d6c81ca7" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293700 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="48841348-a856-4dcc-9687-b987faf399cd" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293709 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="29c08a40-95ad-4783-9cc4-80677e55f751" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293714 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b3a8087-79cb-4f56-bae0-de62d6c81ca7" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293721 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="21614359-ee0c-4cde-b35f-4638cf40c746" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293727 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dab7bc1-1c50-454b-b955-aa4c6fb824e9" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293732 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="48841348-a856-4dcc-9687-b987faf399cd" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293738 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff579236-d055-4e67-864a-40233611afe8" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293744 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b3a8087-79cb-4f56-bae0-de62d6c81ca7" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293751 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="21614359-ee0c-4cde-b35f-4638cf40c746" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293755 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dab7bc1-1c50-454b-b955-aa4c6fb824e9" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293761 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff579236-d055-4e67-864a-40233611afe8" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293766 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="46451788-1ba8-4705-a579-9ab31c9a10b8" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293771 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff579236-d055-4e67-864a-40233611afe8" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293777 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293784 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="46451788-1ba8-4705-a579-9ab31c9a10b8" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293789 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="46451788-1ba8-4705-a579-9ab31c9a10b8" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293795 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="29c08a40-95ad-4783-9cc4-80677e55f751" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293800 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e951a4c-c1a0-4836-8e1e-f3ef9eb605a1" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293805 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dab7bc1-1c50-454b-b955-aa4c6fb824e9" containerName="cleanup" Oct 30 19:15:26.294184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.293811 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="29c08a40-95ad-4783-9cc4-80677e55f751" containerName="cleanup" Oct 30 19:15:26.296958 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.296936 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s" Oct 30 19:15:26.349075 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.349029 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:15:26.349075 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.349029 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:15:26.349347 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.349037 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:15:26.349347 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.349160 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:15:26.400967 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:26.400927 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:15:26.402709 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.402664 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s"] Oct 30 19:15:26.429441 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.429402 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9be28974-d9c4-4009-b515-e9ed39b69772-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s\" (UID: \"9be28974-d9c4-4009-b515-e9ed39b69772\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s" Oct 30 19:15:26.429441 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.429444 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p8zn\" (UniqueName: \"kubernetes.io/projected/9be28974-d9c4-4009-b515-e9ed39b69772-kube-api-access-7p8zn\") pod \"maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s\" (UID: \"9be28974-d9c4-4009-b515-e9ed39b69772\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s" Oct 30 19:15:26.429699 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.429621 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9be28974-d9c4-4009-b515-e9ed39b69772-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s\" (UID: \"9be28974-d9c4-4009-b515-e9ed39b69772\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s" Oct 30 19:15:26.531343 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.531295 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9be28974-d9c4-4009-b515-e9ed39b69772-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s\" (UID: \"9be28974-d9c4-4009-b515-e9ed39b69772\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s" Oct 30 19:15:26.531571 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.531377 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9be28974-d9c4-4009-b515-e9ed39b69772-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s\" (UID: \"9be28974-d9c4-4009-b515-e9ed39b69772\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s" Oct 30 19:15:26.531571 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.531398 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p8zn\" (UniqueName: \"kubernetes.io/projected/9be28974-d9c4-4009-b515-e9ed39b69772-kube-api-access-7p8zn\") pod \"maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s\" (UID: \"9be28974-d9c4-4009-b515-e9ed39b69772\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s" Oct 30 19:15:26.532089 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.532065 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9be28974-d9c4-4009-b515-e9ed39b69772-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s\" (UID: \"9be28974-d9c4-4009-b515-e9ed39b69772\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s" Oct 30 19:15:26.534051 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.534015 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9be28974-d9c4-4009-b515-e9ed39b69772-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s\" (UID: \"9be28974-d9c4-4009-b515-e9ed39b69772\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s" Oct 30 19:15:26.592819 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.592739 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p8zn\" (UniqueName: \"kubernetes.io/projected/9be28974-d9c4-4009-b515-e9ed39b69772-kube-api-access-7p8zn\") pod \"maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s\" (UID: \"9be28974-d9c4-4009-b515-e9ed39b69772\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s" Oct 30 19:15:26.611161 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.611093 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s" Oct 30 19:15:26.993056 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:26.993027 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s"] Oct 30 19:15:26.995367 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:15:26.995334 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be28974_d9c4_4009_b515_e9ed39b69772.slice/crio-b4b1eb31ce445e375a7c4b8faf7228a86d6cf2df3a7b6e1b46e97fc97070d05e WatchSource:0}: Error finding container b4b1eb31ce445e375a7c4b8faf7228a86d6cf2df3a7b6e1b46e97fc97070d05e: Status 404 returned error can't find the container with id b4b1eb31ce445e375a7c4b8faf7228a86d6cf2df3a7b6e1b46e97fc97070d05e Oct 30 19:15:27.491769 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:27.491719 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s" event={"ID":"9be28974-d9c4-4009-b515-e9ed39b69772","Type":"ContainerStarted","Data":"1e8602dc4b5911785e603f4a68bc4881ba042f07e1ec0388b1c62db849be36c8"} Oct 30 19:15:27.492286 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:27.491834 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s" event={"ID":"9be28974-d9c4-4009-b515-e9ed39b69772","Type":"ContainerStarted","Data":"b4b1eb31ce445e375a7c4b8faf7228a86d6cf2df3a7b6e1b46e97fc97070d05e"} Oct 30 19:15:27.611233 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:27.611095 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s" podStartSLOduration=1.611078542 podStartE2EDuration="1.611078542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:27.608504277 +0000 UTC m=+2487.635609239" watchObservedRunningTime="2026-10-30 19:15:27.611078542 +0000 UTC m=+2487.638183503" Oct 30 19:15:28.107642 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.107603 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87bb966f-sj9k2"] Oct 30 19:15:28.108261 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.108237 4531 cpu_manager.go:414] "containerMap: removing container" podUID="1dab7bc1-1c50-454b-b955-aa4c6fb824e9" containerName="cleanup" Oct 30 19:15:28.108261 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.108259 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dab7bc1-1c50-454b-b955-aa4c6fb824e9" containerName="cleanup" Oct 30 19:15:28.108452 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.108279 4531 cpu_manager.go:414] "containerMap: removing container" podUID="21614359-ee0c-4cde-b35f-4638cf40c746" containerName="cleanup" Oct 30 19:15:28.108452 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.108286 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21614359-ee0c-4cde-b35f-4638cf40c746" containerName="cleanup" Oct 30 19:15:28.108452 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.108297 4531 cpu_manager.go:414] "containerMap: removing container" podUID="ff579236-d055-4e67-864a-40233611afe8" containerName="cleanup" Oct 30 19:15:28.108452 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.108304 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff579236-d055-4e67-864a-40233611afe8" containerName="cleanup" Oct 30 19:15:28.108452 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.108340 4531 cpu_manager.go:414] "containerMap: removing container" podUID="29c08a40-95ad-4783-9cc4-80677e55f751" containerName="cleanup" Oct 30 19:15:28.108452 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.108347 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29c08a40-95ad-4783-9cc4-80677e55f751" containerName="cleanup" Oct 30 19:15:28.108452 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.108360 4531 cpu_manager.go:414] "containerMap: removing container" podUID="48841348-a856-4dcc-9687-b987faf399cd" containerName="cleanup" Oct 30 19:15:28.108452 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.108367 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48841348-a856-4dcc-9687-b987faf399cd" containerName="cleanup" Oct 30 19:15:28.108452 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.108379 4531 cpu_manager.go:414] "containerMap: removing container" podUID="9b3a8087-79cb-4f56-bae0-de62d6c81ca7" containerName="cleanup" Oct 30 19:15:28.108452 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.108386 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b3a8087-79cb-4f56-bae0-de62d6c81ca7" containerName="cleanup" Oct 30 19:15:28.108924 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.108509 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b3a8087-79cb-4f56-bae0-de62d6c81ca7" containerName="cleanup" Oct 30 19:15:28.108924 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.108528 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="48841348-a856-4dcc-9687-b987faf399cd" containerName="cleanup" Oct 30 19:15:28.108924 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.108542 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="21614359-ee0c-4cde-b35f-4638cf40c746" containerName="cleanup" Oct 30 19:15:28.112211 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.112181 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bb966f-sj9k2" Oct 30 19:15:28.172320 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.172276 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87bb966f-sj9k2"] Oct 30 19:15:28.252072 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.252031 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54984f9c-9fea-474f-b01e-31054a052892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87bb966f-sj9k2\" (UID: \"54984f9c-9fea-474f-b01e-31054a052892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bb966f-sj9k2" Oct 30 19:15:28.252315 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.252264 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn7tm\" (UniqueName: \"kubernetes.io/projected/54984f9c-9fea-474f-b01e-31054a052892-kube-api-access-dn7tm\") pod \"maas-api-revoke-sub-87bb966f-sj9k2\" (UID: \"54984f9c-9fea-474f-b01e-31054a052892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bb966f-sj9k2" Oct 30 19:15:28.252315 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.252309 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54984f9c-9fea-474f-b01e-31054a052892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87bb966f-sj9k2\" (UID: \"54984f9c-9fea-474f-b01e-31054a052892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bb966f-sj9k2" Oct 30 19:15:28.353020 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.352977 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn7tm\" (UniqueName: \"kubernetes.io/projected/54984f9c-9fea-474f-b01e-31054a052892-kube-api-access-dn7tm\") pod \"maas-api-revoke-sub-87bb966f-sj9k2\" (UID: \"54984f9c-9fea-474f-b01e-31054a052892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bb966f-sj9k2" Oct 30 19:15:28.353020 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.353031 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54984f9c-9fea-474f-b01e-31054a052892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87bb966f-sj9k2\" (UID: \"54984f9c-9fea-474f-b01e-31054a052892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bb966f-sj9k2" Oct 30 19:15:28.353343 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.353096 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54984f9c-9fea-474f-b01e-31054a052892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87bb966f-sj9k2\" (UID: \"54984f9c-9fea-474f-b01e-31054a052892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bb966f-sj9k2" Oct 30 19:15:28.353782 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.353755 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54984f9c-9fea-474f-b01e-31054a052892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87bb966f-sj9k2\" (UID: \"54984f9c-9fea-474f-b01e-31054a052892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bb966f-sj9k2" Oct 30 19:15:28.355870 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.355833 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54984f9c-9fea-474f-b01e-31054a052892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87bb966f-sj9k2\" (UID: \"54984f9c-9fea-474f-b01e-31054a052892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bb966f-sj9k2" Oct 30 19:15:28.415182 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.415124 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn7tm\" (UniqueName: \"kubernetes.io/projected/54984f9c-9fea-474f-b01e-31054a052892-kube-api-access-dn7tm\") pod \"maas-api-revoke-sub-87bb966f-sj9k2\" (UID: \"54984f9c-9fea-474f-b01e-31054a052892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bb966f-sj9k2" Oct 30 19:15:28.430968 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.430932 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bb966f-sj9k2" Oct 30 19:15:28.508226 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.508095 4531 generic.go:356] "Generic (PLEG): container finished" podID="9be28974-d9c4-4009-b515-e9ed39b69772" containerID="1e8602dc4b5911785e603f4a68bc4881ba042f07e1ec0388b1c62db849be36c8" exitCode=0 Oct 30 19:15:28.508676 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.508262 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s" event={"ID":"9be28974-d9c4-4009-b515-e9ed39b69772","Type":"ContainerDied","Data":"1e8602dc4b5911785e603f4a68bc4881ba042f07e1ec0388b1c62db849be36c8"} Oct 30 19:15:28.814210 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:28.814182 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87bb966f-sj9k2"] Oct 30 19:15:28.815976 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:15:28.815942 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54984f9c_9fea_474f_b01e_31054a052892.slice/crio-87f4db70358a3853dc13e80eb84fc1cd5ec2c302b96c8731cec3019a0d74534a WatchSource:0}: Error finding container 87f4db70358a3853dc13e80eb84fc1cd5ec2c302b96c8731cec3019a0d74534a: Status 404 returned error can't find the container with id 87f4db70358a3853dc13e80eb84fc1cd5ec2c302b96c8731cec3019a0d74534a Oct 30 19:15:29.516750 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:29.516711 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bb966f-sj9k2" event={"ID":"54984f9c-9fea-474f-b01e-31054a052892","Type":"ContainerStarted","Data":"cd1d9395e47c54c58fdc906122ca1b835cb6e935366e0abec91b4475e00bfc03"} Oct 30 19:15:29.517193 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:29.516772 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bb966f-sj9k2" event={"ID":"54984f9c-9fea-474f-b01e-31054a052892","Type":"ContainerStarted","Data":"87f4db70358a3853dc13e80eb84fc1cd5ec2c302b96c8731cec3019a0d74534a"} Oct 30 19:15:29.634610 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:29.634552 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bb966f-sj9k2" podStartSLOduration=1.6345368310000001 podStartE2EDuration="1.634536831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:29.632671531 +0000 UTC m=+2489.659776490" watchObservedRunningTime="2026-10-30 19:15:29.634536831 +0000 UTC m=+2489.661641789" Oct 30 19:15:29.671041 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:29.671013 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s" Oct 30 19:15:29.766835 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:29.766738 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9be28974-d9c4-4009-b515-e9ed39b69772-kube-api-access-7p8zn\" (UniqueName: \"kubernetes.io/projected/9be28974-d9c4-4009-b515-e9ed39b69772-kube-api-access-7p8zn\") pod \"9be28974-d9c4-4009-b515-e9ed39b69772\" (UID: \"9be28974-d9c4-4009-b515-e9ed39b69772\") " Oct 30 19:15:29.766997 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:29.766845 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9be28974-d9c4-4009-b515-e9ed39b69772-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9be28974-d9c4-4009-b515-e9ed39b69772-maas-api-xks-ca\") pod \"9be28974-d9c4-4009-b515-e9ed39b69772\" (UID: \"9be28974-d9c4-4009-b515-e9ed39b69772\") " Oct 30 19:15:29.767053 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:29.766992 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9be28974-d9c4-4009-b515-e9ed39b69772-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9be28974-d9c4-4009-b515-e9ed39b69772-maas-api-openshift-ca\") pod \"9be28974-d9c4-4009-b515-e9ed39b69772\" (UID: \"9be28974-d9c4-4009-b515-e9ed39b69772\") " Oct 30 19:15:29.767378 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:29.767350 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9be28974-d9c4-4009-b515-e9ed39b69772-maas-api-openshift-ca" pod "9be28974-d9c4-4009-b515-e9ed39b69772" (UID: "9be28974-d9c4-4009-b515-e9ed39b69772"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:29.769544 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:29.769515 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be28974-d9c4-4009-b515-e9ed39b69772-kube-api-access-7p8zn" pod "9be28974-d9c4-4009-b515-e9ed39b69772" (UID: "9be28974-d9c4-4009-b515-e9ed39b69772"). InnerVolumeSpecName "kube-api-access-7p8zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:29.769544 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:29.769517 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9be28974-d9c4-4009-b515-e9ed39b69772-maas-api-xks-ca" pod "9be28974-d9c4-4009-b515-e9ed39b69772" (UID: "9be28974-d9c4-4009-b515-e9ed39b69772"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:29.868546 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:29.868501 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9be28974-d9c4-4009-b515-e9ed39b69772-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:29.868546 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:29.868537 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9be28974-d9c4-4009-b515-e9ed39b69772-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:29.868546 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:29.868549 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p8zn\" (UniqueName: \"kubernetes.io/projected/9be28974-d9c4-4009-b515-e9ed39b69772-kube-api-access-7p8zn\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:30.523894 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:30.523858 4531 generic.go:356] "Generic (PLEG): container finished" podID="54984f9c-9fea-474f-b01e-31054a052892" containerID="cd1d9395e47c54c58fdc906122ca1b835cb6e935366e0abec91b4475e00bfc03" exitCode=0 Oct 30 19:15:30.524439 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:30.523907 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bb966f-sj9k2" event={"ID":"54984f9c-9fea-474f-b01e-31054a052892","Type":"ContainerDied","Data":"cd1d9395e47c54c58fdc906122ca1b835cb6e935366e0abec91b4475e00bfc03"} Oct 30 19:15:30.525489 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:30.525466 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s" event={"ID":"9be28974-d9c4-4009-b515-e9ed39b69772","Type":"ContainerDied","Data":"b4b1eb31ce445e375a7c4b8faf7228a86d6cf2df3a7b6e1b46e97fc97070d05e"} Oct 30 19:15:30.525582 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:30.525505 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4b1eb31ce445e375a7c4b8faf7228a86d6cf2df3a7b6e1b46e97fc97070d05e" Oct 30 19:15:30.525582 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:30.525475 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s" Oct 30 19:15:31.704467 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:31.704429 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bb966f-sj9k2" Oct 30 19:15:31.891092 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:31.890995 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54984f9c-9fea-474f-b01e-31054a052892-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54984f9c-9fea-474f-b01e-31054a052892-maas-api-openshift-ca\") pod \"54984f9c-9fea-474f-b01e-31054a052892\" (UID: \"54984f9c-9fea-474f-b01e-31054a052892\") " Oct 30 19:15:31.891312 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:31.891221 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54984f9c-9fea-474f-b01e-31054a052892-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54984f9c-9fea-474f-b01e-31054a052892-maas-api-xks-ca\") pod \"54984f9c-9fea-474f-b01e-31054a052892\" (UID: \"54984f9c-9fea-474f-b01e-31054a052892\") " Oct 30 19:15:31.891366 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:31.891309 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54984f9c-9fea-474f-b01e-31054a052892-kube-api-access-dn7tm\" (UniqueName: \"kubernetes.io/projected/54984f9c-9fea-474f-b01e-31054a052892-kube-api-access-dn7tm\") pod \"54984f9c-9fea-474f-b01e-31054a052892\" (UID: \"54984f9c-9fea-474f-b01e-31054a052892\") " Oct 30 19:15:31.891497 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:31.891474 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54984f9c-9fea-474f-b01e-31054a052892-maas-api-openshift-ca" pod "54984f9c-9fea-474f-b01e-31054a052892" (UID: "54984f9c-9fea-474f-b01e-31054a052892"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:31.891679 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:31.891657 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54984f9c-9fea-474f-b01e-31054a052892-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:31.893820 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:31.893787 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54984f9c-9fea-474f-b01e-31054a052892-maas-api-xks-ca" pod "54984f9c-9fea-474f-b01e-31054a052892" (UID: "54984f9c-9fea-474f-b01e-31054a052892"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:31.894015 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:31.893990 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54984f9c-9fea-474f-b01e-31054a052892-kube-api-access-dn7tm" pod "54984f9c-9fea-474f-b01e-31054a052892" (UID: "54984f9c-9fea-474f-b01e-31054a052892"). InnerVolumeSpecName "kube-api-access-dn7tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:31.992943 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:31.992904 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54984f9c-9fea-474f-b01e-31054a052892-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:31.992943 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:31.992937 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn7tm\" (UniqueName: \"kubernetes.io/projected/54984f9c-9fea-474f-b01e-31054a052892-kube-api-access-dn7tm\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:32.539429 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:32.539395 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bb966f-sj9k2" Oct 30 19:15:32.539630 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:32.539424 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bb966f-sj9k2" event={"ID":"54984f9c-9fea-474f-b01e-31054a052892","Type":"ContainerDied","Data":"87f4db70358a3853dc13e80eb84fc1cd5ec2c302b96c8731cec3019a0d74534a"} Oct 30 19:15:32.539630 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:32.539480 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87f4db70358a3853dc13e80eb84fc1cd5ec2c302b96c8731cec3019a0d74534a" Oct 30 19:15:45.254366 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:45.254319 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90a0d000-tms7m"] Oct 30 19:15:45.254851 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:45.254834 4531 cpu_manager.go:414] "containerMap: removing container" podUID="9be28974-d9c4-4009-b515-e9ed39b69772" containerName="revoke-keys" Oct 30 19:15:45.254906 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:45.254853 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9be28974-d9c4-4009-b515-e9ed39b69772" containerName="revoke-keys" Oct 30 19:15:45.254906 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:45.254883 4531 cpu_manager.go:414] "containerMap: removing container" podUID="54984f9c-9fea-474f-b01e-31054a052892" containerName="revoke-keys" Oct 30 19:15:45.254906 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:45.254888 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54984f9c-9fea-474f-b01e-31054a052892" containerName="revoke-keys" Oct 30 19:15:45.255026 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:45.254958 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="54984f9c-9fea-474f-b01e-31054a052892" containerName="revoke-keys" Oct 30 19:15:45.255026 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:45.254971 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9be28974-d9c4-4009-b515-e9ed39b69772" containerName="revoke-keys" Oct 30 19:15:45.258494 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:45.258469 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a0d000-tms7m" Oct 30 19:15:45.310934 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:45.310895 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:15:45.341670 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:45.341628 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b855ed-4915-4eb3-9acf-46a3918dfda9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90a0d000-tms7m\" (UID: \"09b855ed-4915-4eb3-9acf-46a3918dfda9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a0d000-tms7m" Oct 30 19:15:45.341880 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:45.341704 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26mz6\" (UniqueName: \"kubernetes.io/projected/09b855ed-4915-4eb3-9acf-46a3918dfda9-kube-api-access-26mz6\") pod \"maas-api-revoke-sub-90a0d000-tms7m\" (UID: \"09b855ed-4915-4eb3-9acf-46a3918dfda9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a0d000-tms7m" Oct 30 19:15:45.341880 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:45.341822 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b855ed-4915-4eb3-9acf-46a3918dfda9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90a0d000-tms7m\" (UID: \"09b855ed-4915-4eb3-9acf-46a3918dfda9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a0d000-tms7m" Oct 30 19:15:45.360375 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:45.360332 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:15:45.360375 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:45.360365 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:15:45.360626 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:45.360336 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:15:45.363102 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:45.363075 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90a0d000-tms7m"] Oct 30 19:15:45.443040 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:45.443002 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b855ed-4915-4eb3-9acf-46a3918dfda9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90a0d000-tms7m\" (UID: \"09b855ed-4915-4eb3-9acf-46a3918dfda9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a0d000-tms7m" Oct 30 19:15:45.443296 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:45.443070 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26mz6\" (UniqueName: \"kubernetes.io/projected/09b855ed-4915-4eb3-9acf-46a3918dfda9-kube-api-access-26mz6\") pod \"maas-api-revoke-sub-90a0d000-tms7m\" (UID: \"09b855ed-4915-4eb3-9acf-46a3918dfda9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a0d000-tms7m" Oct 30 19:15:45.443296 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:45.443096 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b855ed-4915-4eb3-9acf-46a3918dfda9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90a0d000-tms7m\" (UID: \"09b855ed-4915-4eb3-9acf-46a3918dfda9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a0d000-tms7m" Oct 30 19:15:45.443826 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:45.443804 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b855ed-4915-4eb3-9acf-46a3918dfda9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90a0d000-tms7m\" (UID: \"09b855ed-4915-4eb3-9acf-46a3918dfda9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a0d000-tms7m" Oct 30 19:15:45.445798 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:45.445769 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b855ed-4915-4eb3-9acf-46a3918dfda9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90a0d000-tms7m\" (UID: \"09b855ed-4915-4eb3-9acf-46a3918dfda9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a0d000-tms7m" Oct 30 19:15:45.505392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:45.505304 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26mz6\" (UniqueName: \"kubernetes.io/projected/09b855ed-4915-4eb3-9acf-46a3918dfda9-kube-api-access-26mz6\") pod \"maas-api-revoke-sub-90a0d000-tms7m\" (UID: \"09b855ed-4915-4eb3-9acf-46a3918dfda9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a0d000-tms7m" Oct 30 19:15:45.573599 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:45.573550 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a0d000-tms7m" Oct 30 19:15:45.961116 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:45.961084 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90a0d000-tms7m"] Oct 30 19:15:45.962992 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:15:45.962959 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b855ed_4915_4eb3_9acf_46a3918dfda9.slice/crio-3b536b94b381d8af93f93b9438b0a065b51d615111df0d28bbef8b9c8ff2856b WatchSource:0}: Error finding container 3b536b94b381d8af93f93b9438b0a065b51d615111df0d28bbef8b9c8ff2856b: Status 404 returned error can't find the container with id 3b536b94b381d8af93f93b9438b0a065b51d615111df0d28bbef8b9c8ff2856b Oct 30 19:15:46.625245 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:46.625208 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a0d000-tms7m" event={"ID":"09b855ed-4915-4eb3-9acf-46a3918dfda9","Type":"ContainerStarted","Data":"26d072d07667690d9b81a524fb999f4d5d34994b80369c3208ee1e5a443b2acc"} Oct 30 19:15:46.625695 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:46.625267 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a0d000-tms7m" event={"ID":"09b855ed-4915-4eb3-9acf-46a3918dfda9","Type":"ContainerStarted","Data":"3b536b94b381d8af93f93b9438b0a065b51d615111df0d28bbef8b9c8ff2856b"} Oct 30 19:15:46.744466 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:46.744411 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a0d000-tms7m" podStartSLOduration=1.744395533 podStartE2EDuration="1.744395533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:46.744170371 +0000 UTC m=+2506.771275330" watchObservedRunningTime="2026-10-30 19:15:46.744395533 +0000 UTC m=+2506.771500492" Oct 30 19:15:47.633326 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:47.633288 4531 generic.go:356] "Generic (PLEG): container finished" podID="09b855ed-4915-4eb3-9acf-46a3918dfda9" containerID="26d072d07667690d9b81a524fb999f4d5d34994b80369c3208ee1e5a443b2acc" exitCode=0 Oct 30 19:15:47.633764 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:47.633339 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a0d000-tms7m" event={"ID":"09b855ed-4915-4eb3-9acf-46a3918dfda9","Type":"ContainerDied","Data":"26d072d07667690d9b81a524fb999f4d5d34994b80369c3208ee1e5a443b2acc"} Oct 30 19:15:48.813030 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:48.812995 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a0d000-tms7m" Oct 30 19:15:48.883585 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:48.883539 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09b855ed-4915-4eb3-9acf-46a3918dfda9-kube-api-access-26mz6\" (UniqueName: \"kubernetes.io/projected/09b855ed-4915-4eb3-9acf-46a3918dfda9-kube-api-access-26mz6\") pod \"09b855ed-4915-4eb3-9acf-46a3918dfda9\" (UID: \"09b855ed-4915-4eb3-9acf-46a3918dfda9\") " Oct 30 19:15:48.883769 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:48.883704 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09b855ed-4915-4eb3-9acf-46a3918dfda9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b855ed-4915-4eb3-9acf-46a3918dfda9-maas-api-openshift-ca\") pod \"09b855ed-4915-4eb3-9acf-46a3918dfda9\" (UID: \"09b855ed-4915-4eb3-9acf-46a3918dfda9\") " Oct 30 19:15:48.883769 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:48.883762 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09b855ed-4915-4eb3-9acf-46a3918dfda9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b855ed-4915-4eb3-9acf-46a3918dfda9-maas-api-xks-ca\") pod \"09b855ed-4915-4eb3-9acf-46a3918dfda9\" (UID: \"09b855ed-4915-4eb3-9acf-46a3918dfda9\") " Oct 30 19:15:48.884039 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:48.884014 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09b855ed-4915-4eb3-9acf-46a3918dfda9-maas-api-openshift-ca" pod "09b855ed-4915-4eb3-9acf-46a3918dfda9" (UID: "09b855ed-4915-4eb3-9acf-46a3918dfda9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:48.884269 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:48.884249 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b855ed-4915-4eb3-9acf-46a3918dfda9-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:48.886112 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:48.886083 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09b855ed-4915-4eb3-9acf-46a3918dfda9-maas-api-xks-ca" pod "09b855ed-4915-4eb3-9acf-46a3918dfda9" (UID: "09b855ed-4915-4eb3-9acf-46a3918dfda9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:48.886209 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:48.886160 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09b855ed-4915-4eb3-9acf-46a3918dfda9-kube-api-access-26mz6" pod "09b855ed-4915-4eb3-9acf-46a3918dfda9" (UID: "09b855ed-4915-4eb3-9acf-46a3918dfda9"). InnerVolumeSpecName "kube-api-access-26mz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:48.985349 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:48.985309 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26mz6\" (UniqueName: \"kubernetes.io/projected/09b855ed-4915-4eb3-9acf-46a3918dfda9-kube-api-access-26mz6\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:48.985349 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:48.985348 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b855ed-4915-4eb3-9acf-46a3918dfda9-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:49.648976 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:49.648939 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a0d000-tms7m" Oct 30 19:15:49.648976 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:49.648959 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a0d000-tms7m" event={"ID":"09b855ed-4915-4eb3-9acf-46a3918dfda9","Type":"ContainerDied","Data":"3b536b94b381d8af93f93b9438b0a065b51d615111df0d28bbef8b9c8ff2856b"} Oct 30 19:15:49.649253 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:49.649055 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b536b94b381d8af93f93b9438b0a065b51d615111df0d28bbef8b9c8ff2856b" Oct 30 19:15:56.057749 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:56.057701 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ada8f68-fvbqp"] Oct 30 19:15:56.058209 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:56.058195 4531 cpu_manager.go:414] "containerMap: removing container" podUID="09b855ed-4915-4eb3-9acf-46a3918dfda9" containerName="revoke-keys" Oct 30 19:15:56.058261 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:56.058210 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09b855ed-4915-4eb3-9acf-46a3918dfda9" containerName="revoke-keys" Oct 30 19:15:56.058295 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:56.058290 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="09b855ed-4915-4eb3-9acf-46a3918dfda9" containerName="revoke-keys" Oct 30 19:15:56.061676 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:56.061654 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8f68-fvbqp" Oct 30 19:15:56.114420 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:56.114378 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:15:56.114420 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:56.114402 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:15:56.114644 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:56.114464 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:15:56.164074 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:56.164030 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:15:56.166813 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:56.166777 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ada8f68-fvbqp"] Oct 30 19:15:56.167334 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:56.167308 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gjlq\" (UniqueName: \"kubernetes.io/projected/c8634059-e325-4cb7-95e2-a443a9ee8757-kube-api-access-9gjlq\") pod \"maas-api-revoke-sub-5ada8f68-fvbqp\" (UID: \"c8634059-e325-4cb7-95e2-a443a9ee8757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8f68-fvbqp" Oct 30 19:15:56.167405 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:56.167364 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8634059-e325-4cb7-95e2-a443a9ee8757-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ada8f68-fvbqp\" (UID: \"c8634059-e325-4cb7-95e2-a443a9ee8757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8f68-fvbqp" Oct 30 19:15:56.167479 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:56.167463 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8634059-e325-4cb7-95e2-a443a9ee8757-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ada8f68-fvbqp\" (UID: \"c8634059-e325-4cb7-95e2-a443a9ee8757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8f68-fvbqp" Oct 30 19:15:56.268752 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:56.268687 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gjlq\" (UniqueName: \"kubernetes.io/projected/c8634059-e325-4cb7-95e2-a443a9ee8757-kube-api-access-9gjlq\") pod \"maas-api-revoke-sub-5ada8f68-fvbqp\" (UID: \"c8634059-e325-4cb7-95e2-a443a9ee8757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8f68-fvbqp" Oct 30 19:15:56.268985 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:56.268781 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8634059-e325-4cb7-95e2-a443a9ee8757-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ada8f68-fvbqp\" (UID: \"c8634059-e325-4cb7-95e2-a443a9ee8757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8f68-fvbqp" Oct 30 19:15:56.268985 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:56.268857 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8634059-e325-4cb7-95e2-a443a9ee8757-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ada8f68-fvbqp\" (UID: \"c8634059-e325-4cb7-95e2-a443a9ee8757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8f68-fvbqp" Oct 30 19:15:56.269513 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:56.269487 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8634059-e325-4cb7-95e2-a443a9ee8757-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ada8f68-fvbqp\" (UID: \"c8634059-e325-4cb7-95e2-a443a9ee8757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8f68-fvbqp" Oct 30 19:15:56.271767 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:56.271727 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8634059-e325-4cb7-95e2-a443a9ee8757-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ada8f68-fvbqp\" (UID: \"c8634059-e325-4cb7-95e2-a443a9ee8757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8f68-fvbqp" Oct 30 19:15:56.330411 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:56.330332 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gjlq\" (UniqueName: \"kubernetes.io/projected/c8634059-e325-4cb7-95e2-a443a9ee8757-kube-api-access-9gjlq\") pod \"maas-api-revoke-sub-5ada8f68-fvbqp\" (UID: \"c8634059-e325-4cb7-95e2-a443a9ee8757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8f68-fvbqp" Oct 30 19:15:56.376920 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:56.376875 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8f68-fvbqp" Oct 30 19:15:56.402090 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:56.402050 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:15:56.762173 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:56.762107 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ada8f68-fvbqp"] Oct 30 19:15:56.764338 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:15:56.764307 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8634059_e325_4cb7_95e2_a443a9ee8757.slice/crio-4393892475a932e6f29bbfac96a4a95a04321ec64a60bd7fef0279beee632c2a WatchSource:0}: Error finding container 4393892475a932e6f29bbfac96a4a95a04321ec64a60bd7fef0279beee632c2a: Status 404 returned error can't find the container with id 4393892475a932e6f29bbfac96a4a95a04321ec64a60bd7fef0279beee632c2a Oct 30 19:15:57.700746 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:57.700698 4531 generic.go:356] "Generic (PLEG): container finished" podID="c8634059-e325-4cb7-95e2-a443a9ee8757" containerID="1d8364e6e0e901b7467ac75f956092add540bf6602c979ecad8b87419abd93b3" exitCode=0 Oct 30 19:15:57.701204 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:57.700832 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8f68-fvbqp" event={"ID":"c8634059-e325-4cb7-95e2-a443a9ee8757","Type":"ContainerDied","Data":"1d8364e6e0e901b7467ac75f956092add540bf6602c979ecad8b87419abd93b3"} Oct 30 19:15:57.701204 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:57.700897 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8f68-fvbqp" event={"ID":"c8634059-e325-4cb7-95e2-a443a9ee8757","Type":"ContainerStarted","Data":"4393892475a932e6f29bbfac96a4a95a04321ec64a60bd7fef0279beee632c2a"} Oct 30 19:15:57.898272 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:57.898171 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs"] Oct 30 19:15:57.898464 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:57.898411 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs" podUID="a9ace4a6-85c4-477a-99d5-0995681b0bb7" containerName="payload-processing" containerID="cri-o://d9d653d0c5d65caafa98474b437654c31be00d48a751629bcd06040c0aa6939d" gracePeriod=30 Oct 30 19:15:57.900862 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:57.900830 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl"] Oct 30 19:15:57.901077 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:57.901053 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl" podUID="194c0736-90dd-41b0-b316-9858113b2477" containerName="payload-pre-processing" containerID="cri-o://ce5d5826f2eff92976a4c76fc110ee9b7407386c0a4f043ffa2dac0551b42b31" gracePeriod=30 Oct 30 19:15:58.203811 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.203780 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl" Oct 30 19:15:58.208072 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.208042 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs" Oct 30 19:15:58.396591 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.396557 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9ace4a6-85c4-477a-99d5-0995681b0bb7-kube-api-access-sx562\" (UniqueName: \"kubernetes.io/projected/a9ace4a6-85c4-477a-99d5-0995681b0bb7-kube-api-access-sx562\") pod \"a9ace4a6-85c4-477a-99d5-0995681b0bb7\" (UID: \"a9ace4a6-85c4-477a-99d5-0995681b0bb7\") " Oct 30 19:15:58.396819 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.396620 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/194c0736-90dd-41b0-b316-9858113b2477-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/194c0736-90dd-41b0-b316-9858113b2477-plugins-config-volume\") pod \"194c0736-90dd-41b0-b316-9858113b2477\" (UID: \"194c0736-90dd-41b0-b316-9858113b2477\") " Oct 30 19:15:58.396819 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.396780 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/194c0736-90dd-41b0-b316-9858113b2477-kube-api-access-jssqx\" (UniqueName: \"kubernetes.io/projected/194c0736-90dd-41b0-b316-9858113b2477-kube-api-access-jssqx\") pod \"194c0736-90dd-41b0-b316-9858113b2477\" (UID: \"194c0736-90dd-41b0-b316-9858113b2477\") " Oct 30 19:15:58.396959 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.396835 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9ace4a6-85c4-477a-99d5-0995681b0bb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ace4a6-85c4-477a-99d5-0995681b0bb7-plugins-config-volume\") pod \"a9ace4a6-85c4-477a-99d5-0995681b0bb7\" (UID: \"a9ace4a6-85c4-477a-99d5-0995681b0bb7\") " Oct 30 19:15:58.397306 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.397267 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/194c0736-90dd-41b0-b316-9858113b2477-plugins-config-volume" pod "194c0736-90dd-41b0-b316-9858113b2477" (UID: "194c0736-90dd-41b0-b316-9858113b2477"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:58.397407 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.397369 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ace4a6-85c4-477a-99d5-0995681b0bb7-plugins-config-volume" pod "a9ace4a6-85c4-477a-99d5-0995681b0bb7" (UID: "a9ace4a6-85c4-477a-99d5-0995681b0bb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:58.399834 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.399788 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ace4a6-85c4-477a-99d5-0995681b0bb7-kube-api-access-sx562" pod "a9ace4a6-85c4-477a-99d5-0995681b0bb7" (UID: "a9ace4a6-85c4-477a-99d5-0995681b0bb7"). InnerVolumeSpecName "kube-api-access-sx562". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:58.399990 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.399956 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194c0736-90dd-41b0-b316-9858113b2477-kube-api-access-jssqx" pod "194c0736-90dd-41b0-b316-9858113b2477" (UID: "194c0736-90dd-41b0-b316-9858113b2477"). InnerVolumeSpecName "kube-api-access-jssqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:58.498040 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.498004 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jssqx\" (UniqueName: \"kubernetes.io/projected/194c0736-90dd-41b0-b316-9858113b2477-kube-api-access-jssqx\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:58.498040 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.498041 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ace4a6-85c4-477a-99d5-0995681b0bb7-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:58.498267 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.498050 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx562\" (UniqueName: \"kubernetes.io/projected/a9ace4a6-85c4-477a-99d5-0995681b0bb7-kube-api-access-sx562\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:58.498267 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.498059 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/194c0736-90dd-41b0-b316-9858113b2477-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:58.709218 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.709183 4531 generic.go:356] "Generic (PLEG): container finished" podID="194c0736-90dd-41b0-b316-9858113b2477" containerID="ce5d5826f2eff92976a4c76fc110ee9b7407386c0a4f043ffa2dac0551b42b31" exitCode=0 Oct 30 19:15:58.709761 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.709250 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl" Oct 30 19:15:58.709761 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.709281 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl" event={"ID":"194c0736-90dd-41b0-b316-9858113b2477","Type":"ContainerDied","Data":"ce5d5826f2eff92976a4c76fc110ee9b7407386c0a4f043ffa2dac0551b42b31"} Oct 30 19:15:58.709761 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.709346 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl" event={"ID":"194c0736-90dd-41b0-b316-9858113b2477","Type":"ContainerDied","Data":"ec70e644050ada76b1ef9c73b4379f43df456ed1e769be08bb24fc3a29a8a0f1"} Oct 30 19:15:58.709761 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.709374 4531 scope.go:122] "RemoveContainer" containerID="ce5d5826f2eff92976a4c76fc110ee9b7407386c0a4f043ffa2dac0551b42b31" Oct 30 19:15:58.710881 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.710858 4531 generic.go:356] "Generic (PLEG): container finished" podID="a9ace4a6-85c4-477a-99d5-0995681b0bb7" containerID="d9d653d0c5d65caafa98474b437654c31be00d48a751629bcd06040c0aa6939d" exitCode=0 Oct 30 19:15:58.710991 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.710912 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs" Oct 30 19:15:58.710991 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.710920 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs" event={"ID":"a9ace4a6-85c4-477a-99d5-0995681b0bb7","Type":"ContainerDied","Data":"d9d653d0c5d65caafa98474b437654c31be00d48a751629bcd06040c0aa6939d"} Oct 30 19:15:58.710991 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.710985 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs" event={"ID":"a9ace4a6-85c4-477a-99d5-0995681b0bb7","Type":"ContainerDied","Data":"c0373d6f71ad71d9417d6bc7540e928134e29ee48df658c1df80529af8fc8f27"} Oct 30 19:15:58.729231 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.729196 4531 scope.go:122] "RemoveContainer" containerID="ce5d5826f2eff92976a4c76fc110ee9b7407386c0a4f043ffa2dac0551b42b31" Oct 30 19:15:58.729799 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:58.729763 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce5d5826f2eff92976a4c76fc110ee9b7407386c0a4f043ffa2dac0551b42b31\": container with ID starting with ce5d5826f2eff92976a4c76fc110ee9b7407386c0a4f043ffa2dac0551b42b31 not found: ID does not exist" containerID="ce5d5826f2eff92976a4c76fc110ee9b7407386c0a4f043ffa2dac0551b42b31" Oct 30 19:15:58.729927 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.729812 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce5d5826f2eff92976a4c76fc110ee9b7407386c0a4f043ffa2dac0551b42b31"} err="failed to get container status \"ce5d5826f2eff92976a4c76fc110ee9b7407386c0a4f043ffa2dac0551b42b31\": rpc error: code = NotFound desc = could not find container \"ce5d5826f2eff92976a4c76fc110ee9b7407386c0a4f043ffa2dac0551b42b31\": container with ID starting with ce5d5826f2eff92976a4c76fc110ee9b7407386c0a4f043ffa2dac0551b42b31 not found: ID does not exist" Oct 30 19:15:58.729927 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.729839 4531 scope.go:122] "RemoveContainer" containerID="d9d653d0c5d65caafa98474b437654c31be00d48a751629bcd06040c0aa6939d" Oct 30 19:15:58.740760 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.740734 4531 scope.go:122] "RemoveContainer" containerID="d9d653d0c5d65caafa98474b437654c31be00d48a751629bcd06040c0aa6939d" Oct 30 19:15:58.741174 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:15:58.741147 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9d653d0c5d65caafa98474b437654c31be00d48a751629bcd06040c0aa6939d\": container with ID starting with d9d653d0c5d65caafa98474b437654c31be00d48a751629bcd06040c0aa6939d not found: ID does not exist" containerID="d9d653d0c5d65caafa98474b437654c31be00d48a751629bcd06040c0aa6939d" Oct 30 19:15:58.741243 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.741187 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9d653d0c5d65caafa98474b437654c31be00d48a751629bcd06040c0aa6939d"} err="failed to get container status \"d9d653d0c5d65caafa98474b437654c31be00d48a751629bcd06040c0aa6939d\": rpc error: code = NotFound desc = could not find container \"d9d653d0c5d65caafa98474b437654c31be00d48a751629bcd06040c0aa6939d\": container with ID starting with d9d653d0c5d65caafa98474b437654c31be00d48a751629bcd06040c0aa6939d not found: ID does not exist" Oct 30 19:15:58.867690 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.867650 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8f68-fvbqp" Oct 30 19:15:58.878523 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.878484 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl"] Oct 30 19:15:58.880994 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.880960 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0c9a2b-995cf9cf9-qhnzl"] Oct 30 19:15:58.990452 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.990410 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs"] Oct 30 19:15:58.997495 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:58.997459 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0c9a2b-c5d9f8b84-f6qrs"] Oct 30 19:15:59.004149 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:59.004100 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8634059-e325-4cb7-95e2-a443a9ee8757-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8634059-e325-4cb7-95e2-a443a9ee8757-maas-api-xks-ca\") pod \"c8634059-e325-4cb7-95e2-a443a9ee8757\" (UID: \"c8634059-e325-4cb7-95e2-a443a9ee8757\") " Oct 30 19:15:59.004310 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:59.004189 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8634059-e325-4cb7-95e2-a443a9ee8757-kube-api-access-9gjlq\" (UniqueName: \"kubernetes.io/projected/c8634059-e325-4cb7-95e2-a443a9ee8757-kube-api-access-9gjlq\") pod \"c8634059-e325-4cb7-95e2-a443a9ee8757\" (UID: \"c8634059-e325-4cb7-95e2-a443a9ee8757\") " Oct 30 19:15:59.004310 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:59.004216 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8634059-e325-4cb7-95e2-a443a9ee8757-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8634059-e325-4cb7-95e2-a443a9ee8757-maas-api-openshift-ca\") pod \"c8634059-e325-4cb7-95e2-a443a9ee8757\" (UID: \"c8634059-e325-4cb7-95e2-a443a9ee8757\") " Oct 30 19:15:59.004731 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:59.004693 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8634059-e325-4cb7-95e2-a443a9ee8757-maas-api-openshift-ca" pod "c8634059-e325-4cb7-95e2-a443a9ee8757" (UID: "c8634059-e325-4cb7-95e2-a443a9ee8757"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:59.006706 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:59.006672 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8634059-e325-4cb7-95e2-a443a9ee8757-maas-api-xks-ca" pod "c8634059-e325-4cb7-95e2-a443a9ee8757" (UID: "c8634059-e325-4cb7-95e2-a443a9ee8757"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:59.006706 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:59.006682 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8634059-e325-4cb7-95e2-a443a9ee8757-kube-api-access-9gjlq" pod "c8634059-e325-4cb7-95e2-a443a9ee8757" (UID: "c8634059-e325-4cb7-95e2-a443a9ee8757"). InnerVolumeSpecName "kube-api-access-9gjlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:59.105500 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:59.105409 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8634059-e325-4cb7-95e2-a443a9ee8757-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:59.105500 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:59.105445 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gjlq\" (UniqueName: \"kubernetes.io/projected/c8634059-e325-4cb7-95e2-a443a9ee8757-kube-api-access-9gjlq\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:59.105500 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:59.105457 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8634059-e325-4cb7-95e2-a443a9ee8757-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:15:59.574995 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:59.574953 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd"] Oct 30 19:15:59.578101 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:59.578065 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0c9a2b-6bfpd"] Oct 30 19:15:59.721760 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:59.721732 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8f68-fvbqp" Oct 30 19:15:59.722163 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:59.721773 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ada8f68-fvbqp" event={"ID":"c8634059-e325-4cb7-95e2-a443a9ee8757","Type":"ContainerDied","Data":"4393892475a932e6f29bbfac96a4a95a04321ec64a60bd7fef0279beee632c2a"} Oct 30 19:15:59.722163 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:15:59.721820 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4393892475a932e6f29bbfac96a4a95a04321ec64a60bd7fef0279beee632c2a" Oct 30 19:16:00.098447 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:00.098412 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="194c0736-90dd-41b0-b316-9858113b2477" path="/var/lib/kubelet/pods/194c0736-90dd-41b0-b316-9858113b2477/volumes" Oct 30 19:16:00.098901 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:00.098885 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ef6f5e4-3502-42f1-a565-a19a131e2a2a" path="/var/lib/kubelet/pods/6ef6f5e4-3502-42f1-a565-a19a131e2a2a/volumes" Oct 30 19:16:00.099274 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:00.099260 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9ace4a6-85c4-477a-99d5-0995681b0bb7" path="/var/lib/kubelet/pods/a9ace4a6-85c4-477a-99d5-0995681b0bb7/volumes" Oct 30 19:16:01.898026 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:01.897987 4531 scope.go:122] "RemoveContainer" containerID="785dfcee4fd680d9b44438cf0f14da3639db220daf24f1ceb59936f0072960a9" Oct 30 19:16:05.092325 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:05.092284 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6"] Oct 30 19:16:12.287607 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.287567 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e7d337-8c827"] Oct 30 19:16:12.288011 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.287969 4531 cpu_manager.go:414] "containerMap: removing container" podUID="a9ace4a6-85c4-477a-99d5-0995681b0bb7" containerName="payload-processing" Oct 30 19:16:12.288011 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.287980 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9ace4a6-85c4-477a-99d5-0995681b0bb7" containerName="payload-processing" Oct 30 19:16:12.288011 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.287990 4531 cpu_manager.go:414] "containerMap: removing container" podUID="194c0736-90dd-41b0-b316-9858113b2477" containerName="payload-pre-processing" Oct 30 19:16:12.288011 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.287996 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="194c0736-90dd-41b0-b316-9858113b2477" containerName="payload-pre-processing" Oct 30 19:16:12.288174 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.288030 4531 cpu_manager.go:414] "containerMap: removing container" podUID="c8634059-e325-4cb7-95e2-a443a9ee8757" containerName="revoke-keys" Oct 30 19:16:12.288174 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.288035 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8634059-e325-4cb7-95e2-a443a9ee8757" containerName="revoke-keys" Oct 30 19:16:12.288174 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.288100 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8634059-e325-4cb7-95e2-a443a9ee8757" containerName="revoke-keys" Oct 30 19:16:12.288174 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.288107 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9ace4a6-85c4-477a-99d5-0995681b0bb7" containerName="payload-processing" Oct 30 19:16:12.288174 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.288115 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="194c0736-90dd-41b0-b316-9858113b2477" containerName="payload-pre-processing" Oct 30 19:16:12.300185 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.300141 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e7d337-8c827" Oct 30 19:16:12.332015 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.331981 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bndhf\" (UniqueName: \"kubernetes.io/projected/587afca3-fc5e-4ddc-b0d3-d49713d4257d-kube-api-access-bndhf\") pod \"maas-api-revoke-sub-c2e7d337-8c827\" (UID: \"587afca3-fc5e-4ddc-b0d3-d49713d4257d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e7d337-8c827" Oct 30 19:16:12.332015 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.332027 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/587afca3-fc5e-4ddc-b0d3-d49713d4257d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2e7d337-8c827\" (UID: \"587afca3-fc5e-4ddc-b0d3-d49713d4257d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e7d337-8c827" Oct 30 19:16:12.332320 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.332056 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/587afca3-fc5e-4ddc-b0d3-d49713d4257d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2e7d337-8c827\" (UID: \"587afca3-fc5e-4ddc-b0d3-d49713d4257d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e7d337-8c827" Oct 30 19:16:12.352597 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.352552 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e7d337-8c827"] Oct 30 19:16:12.398323 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.398285 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:16:12.398323 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.398289 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:16:12.398323 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.398314 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:16:12.404916 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.404893 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:16:12.433093 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.433044 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bndhf\" (UniqueName: \"kubernetes.io/projected/587afca3-fc5e-4ddc-b0d3-d49713d4257d-kube-api-access-bndhf\") pod \"maas-api-revoke-sub-c2e7d337-8c827\" (UID: \"587afca3-fc5e-4ddc-b0d3-d49713d4257d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e7d337-8c827" Oct 30 19:16:12.433355 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.433104 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/587afca3-fc5e-4ddc-b0d3-d49713d4257d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2e7d337-8c827\" (UID: \"587afca3-fc5e-4ddc-b0d3-d49713d4257d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e7d337-8c827" Oct 30 19:16:12.433355 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.433181 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/587afca3-fc5e-4ddc-b0d3-d49713d4257d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2e7d337-8c827\" (UID: \"587afca3-fc5e-4ddc-b0d3-d49713d4257d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e7d337-8c827" Oct 30 19:16:12.433818 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.433794 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/587afca3-fc5e-4ddc-b0d3-d49713d4257d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2e7d337-8c827\" (UID: \"587afca3-fc5e-4ddc-b0d3-d49713d4257d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e7d337-8c827" Oct 30 19:16:12.435822 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.435801 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/587afca3-fc5e-4ddc-b0d3-d49713d4257d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2e7d337-8c827\" (UID: \"587afca3-fc5e-4ddc-b0d3-d49713d4257d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e7d337-8c827" Oct 30 19:16:12.494552 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.494508 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bndhf\" (UniqueName: \"kubernetes.io/projected/587afca3-fc5e-4ddc-b0d3-d49713d4257d-kube-api-access-bndhf\") pod \"maas-api-revoke-sub-c2e7d337-8c827\" (UID: \"587afca3-fc5e-4ddc-b0d3-d49713d4257d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e7d337-8c827" Oct 30 19:16:12.613093 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.613001 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e7d337-8c827" Oct 30 19:16:12.996455 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:12.996428 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e7d337-8c827"] Oct 30 19:16:12.998617 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:16:12.998581 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587afca3_fc5e_4ddc_b0d3_d49713d4257d.slice/crio-d860b2b6f214d7a3b606376b16e072d3b07ed39bf53e61840b12094cd662bc99 WatchSource:0}: Error finding container d860b2b6f214d7a3b606376b16e072d3b07ed39bf53e61840b12094cd662bc99: Status 404 returned error can't find the container with id d860b2b6f214d7a3b606376b16e072d3b07ed39bf53e61840b12094cd662bc99 Oct 30 19:16:13.810354 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:13.810313 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e7d337-8c827" event={"ID":"587afca3-fc5e-4ddc-b0d3-d49713d4257d","Type":"ContainerStarted","Data":"e92d8b69c7d1999443e2fc8e63eb6d3fb45cc933f75a295d31e0621874d142cc"} Oct 30 19:16:13.810761 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:13.810384 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e7d337-8c827" event={"ID":"587afca3-fc5e-4ddc-b0d3-d49713d4257d","Type":"ContainerStarted","Data":"d860b2b6f214d7a3b606376b16e072d3b07ed39bf53e61840b12094cd662bc99"} Oct 30 19:16:13.928499 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:13.928446 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e7d337-8c827" podStartSLOduration=1.9284314569999998 podStartE2EDuration="1.928431457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:13.92586152 +0000 UTC m=+2533.952966479" watchObservedRunningTime="2026-10-30 19:16:13.928431457 +0000 UTC m=+2533.955536416" Oct 30 19:16:14.818026 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:14.817990 4531 generic.go:356] "Generic (PLEG): container finished" podID="587afca3-fc5e-4ddc-b0d3-d49713d4257d" containerID="e92d8b69c7d1999443e2fc8e63eb6d3fb45cc933f75a295d31e0621874d142cc" exitCode=0 Oct 30 19:16:14.818490 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:14.818087 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e7d337-8c827" event={"ID":"587afca3-fc5e-4ddc-b0d3-d49713d4257d","Type":"ContainerDied","Data":"e92d8b69c7d1999443e2fc8e63eb6d3fb45cc933f75a295d31e0621874d142cc"} Oct 30 19:16:15.987327 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:15.987294 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e7d337-8c827" Oct 30 19:16:16.073805 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:16.073766 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/587afca3-fc5e-4ddc-b0d3-d49713d4257d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/587afca3-fc5e-4ddc-b0d3-d49713d4257d-maas-api-openshift-ca\") pod \"587afca3-fc5e-4ddc-b0d3-d49713d4257d\" (UID: \"587afca3-fc5e-4ddc-b0d3-d49713d4257d\") " Oct 30 19:16:16.074014 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:16.073817 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/587afca3-fc5e-4ddc-b0d3-d49713d4257d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/587afca3-fc5e-4ddc-b0d3-d49713d4257d-maas-api-xks-ca\") pod \"587afca3-fc5e-4ddc-b0d3-d49713d4257d\" (UID: \"587afca3-fc5e-4ddc-b0d3-d49713d4257d\") " Oct 30 19:16:16.074014 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:16.073946 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/587afca3-fc5e-4ddc-b0d3-d49713d4257d-kube-api-access-bndhf\" (UniqueName: \"kubernetes.io/projected/587afca3-fc5e-4ddc-b0d3-d49713d4257d-kube-api-access-bndhf\") pod \"587afca3-fc5e-4ddc-b0d3-d49713d4257d\" (UID: \"587afca3-fc5e-4ddc-b0d3-d49713d4257d\") " Oct 30 19:16:16.074395 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:16.074370 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587afca3-fc5e-4ddc-b0d3-d49713d4257d-maas-api-openshift-ca" pod "587afca3-fc5e-4ddc-b0d3-d49713d4257d" (UID: "587afca3-fc5e-4ddc-b0d3-d49713d4257d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:16.076340 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:16.076262 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/587afca3-fc5e-4ddc-b0d3-d49713d4257d-maas-api-xks-ca" pod "587afca3-fc5e-4ddc-b0d3-d49713d4257d" (UID: "587afca3-fc5e-4ddc-b0d3-d49713d4257d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:16:16.076340 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:16.076282 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587afca3-fc5e-4ddc-b0d3-d49713d4257d-kube-api-access-bndhf" pod "587afca3-fc5e-4ddc-b0d3-d49713d4257d" (UID: "587afca3-fc5e-4ddc-b0d3-d49713d4257d"). InnerVolumeSpecName "kube-api-access-bndhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:16.175436 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:16.175403 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bndhf\" (UniqueName: \"kubernetes.io/projected/587afca3-fc5e-4ddc-b0d3-d49713d4257d-kube-api-access-bndhf\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:16:16.175436 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:16.175435 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/587afca3-fc5e-4ddc-b0d3-d49713d4257d-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:16:16.175436 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:16.175445 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/587afca3-fc5e-4ddc-b0d3-d49713d4257d-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:16:16.831969 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:16.831931 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e7d337-8c827" Oct 30 19:16:16.832192 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:16.831933 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e7d337-8c827" event={"ID":"587afca3-fc5e-4ddc-b0d3-d49713d4257d","Type":"ContainerDied","Data":"d860b2b6f214d7a3b606376b16e072d3b07ed39bf53e61840b12094cd662bc99"} Oct 30 19:16:16.832192 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:16.832056 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d860b2b6f214d7a3b606376b16e072d3b07ed39bf53e61840b12094cd662bc99" Oct 30 19:16:20.113476 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.113432 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" podUID="6796af55-3e88-4dec-9b8a-9232d8c10e16" containerName="main" containerID="cri-o://156ba02f779d4dbc5f228b816a66fe061680bc56db81bb5c0e206c5b513d18d1" gracePeriod=45 Oct 30 19:16:20.396364 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.396333 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" Oct 30 19:16:20.523110 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.522048 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-home\" (UniqueName: \"kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-home\") pod \"6796af55-3e88-4dec-9b8a-9232d8c10e16\" (UID: \"6796af55-3e88-4dec-9b8a-9232d8c10e16\") " Oct 30 19:16:20.523110 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.522171 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-dshm\") pod \"6796af55-3e88-4dec-9b8a-9232d8c10e16\" (UID: \"6796af55-3e88-4dec-9b8a-9232d8c10e16\") " Oct 30 19:16:20.523110 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.522251 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-tmp-dir\") pod \"6796af55-3e88-4dec-9b8a-9232d8c10e16\" (UID: \"6796af55-3e88-4dec-9b8a-9232d8c10e16\") " Oct 30 19:16:20.523110 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.522315 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6796af55-3e88-4dec-9b8a-9232d8c10e16-tls-certs\" (UniqueName: \"kubernetes.io/secret/6796af55-3e88-4dec-9b8a-9232d8c10e16-tls-certs\") pod \"6796af55-3e88-4dec-9b8a-9232d8c10e16\" (UID: \"6796af55-3e88-4dec-9b8a-9232d8c10e16\") " Oct 30 19:16:20.523110 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.522401 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-model-cache\") pod \"6796af55-3e88-4dec-9b8a-9232d8c10e16\" (UID: \"6796af55-3e88-4dec-9b8a-9232d8c10e16\") " Oct 30 19:16:20.523110 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.522458 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6796af55-3e88-4dec-9b8a-9232d8c10e16-kube-api-access-nv4gp\" (UniqueName: \"kubernetes.io/projected/6796af55-3e88-4dec-9b8a-9232d8c10e16-kube-api-access-nv4gp\") pod \"6796af55-3e88-4dec-9b8a-9232d8c10e16\" (UID: \"6796af55-3e88-4dec-9b8a-9232d8c10e16\") " Oct 30 19:16:20.535686 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.526465 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-home" pod "6796af55-3e88-4dec-9b8a-9232d8c10e16" (UID: "6796af55-3e88-4dec-9b8a-9232d8c10e16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:20.535686 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.526480 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-tmp-dir" pod "6796af55-3e88-4dec-9b8a-9232d8c10e16" (UID: "6796af55-3e88-4dec-9b8a-9232d8c10e16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:20.535686 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.526707 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-model-cache" pod "6796af55-3e88-4dec-9b8a-9232d8c10e16" (UID: "6796af55-3e88-4dec-9b8a-9232d8c10e16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:20.535686 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.527434 4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-tmp-dir\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:16:20.535686 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.527458 4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-model-cache\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:16:20.535686 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.527472 4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-home\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:16:20.535686 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.529461 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6796af55-3e88-4dec-9b8a-9232d8c10e16-tls-certs" pod "6796af55-3e88-4dec-9b8a-9232d8c10e16" (UID: "6796af55-3e88-4dec-9b8a-9232d8c10e16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:16:20.535686 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.529556 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6796af55-3e88-4dec-9b8a-9232d8c10e16-kube-api-access-nv4gp" pod "6796af55-3e88-4dec-9b8a-9232d8c10e16" (UID: "6796af55-3e88-4dec-9b8a-9232d8c10e16"). InnerVolumeSpecName "kube-api-access-nv4gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:20.537721 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.537682 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-dshm" pod "6796af55-3e88-4dec-9b8a-9232d8c10e16" (UID: "6796af55-3e88-4dec-9b8a-9232d8c10e16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:20.628448 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.628361 4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6796af55-3e88-4dec-9b8a-9232d8c10e16-tls-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:16:20.628448 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.628393 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv4gp\" (UniqueName: \"kubernetes.io/projected/6796af55-3e88-4dec-9b8a-9232d8c10e16-kube-api-access-nv4gp\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:16:20.628448 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.628405 4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6796af55-3e88-4dec-9b8a-9232d8c10e16-dshm\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:16:20.858609 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.858572 4531 generic.go:356] "Generic (PLEG): container finished" podID="6796af55-3e88-4dec-9b8a-9232d8c10e16" containerID="156ba02f779d4dbc5f228b816a66fe061680bc56db81bb5c0e206c5b513d18d1" exitCode=0 Oct 30 19:16:20.858798 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.858617 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" Oct 30 19:16:20.858798 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.858669 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" event={"ID":"6796af55-3e88-4dec-9b8a-9232d8c10e16","Type":"ContainerDied","Data":"156ba02f779d4dbc5f228b816a66fe061680bc56db81bb5c0e206c5b513d18d1"} Oct 30 19:16:20.858798 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.858722 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6" event={"ID":"6796af55-3e88-4dec-9b8a-9232d8c10e16","Type":"ContainerDied","Data":"a444ab54a8653fc70746951da64c589324e197e03456e80ac14e04a80f0dced9"} Oct 30 19:16:20.858798 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.858746 4531 scope.go:122] "RemoveContainer" containerID="156ba02f779d4dbc5f228b816a66fe061680bc56db81bb5c0e206c5b513d18d1" Oct 30 19:16:20.868768 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.868745 4531 scope.go:122] "RemoveContainer" containerID="156ba02f779d4dbc5f228b816a66fe061680bc56db81bb5c0e206c5b513d18d1" Oct 30 19:16:20.869078 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:16:20.869058 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"156ba02f779d4dbc5f228b816a66fe061680bc56db81bb5c0e206c5b513d18d1\": container with ID starting with 156ba02f779d4dbc5f228b816a66fe061680bc56db81bb5c0e206c5b513d18d1 not found: ID does not exist" containerID="156ba02f779d4dbc5f228b816a66fe061680bc56db81bb5c0e206c5b513d18d1" Oct 30 19:16:20.869196 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:20.869088 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"156ba02f779d4dbc5f228b816a66fe061680bc56db81bb5c0e206c5b513d18d1"} err="failed to get container status \"156ba02f779d4dbc5f228b816a66fe061680bc56db81bb5c0e206c5b513d18d1\": rpc error: code = NotFound desc = could not find container \"156ba02f779d4dbc5f228b816a66fe061680bc56db81bb5c0e206c5b513d18d1\": container with ID starting with 156ba02f779d4dbc5f228b816a66fe061680bc56db81bb5c0e206c5b513d18d1 not found: ID does not exist" Oct 30 19:16:21.027752 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:21.027719 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6"] Oct 30 19:16:21.076863 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:21.076822 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-0c9a2b/ipp-route-0c9a2b-kserve-76847b9cbf-qbqk6"] Oct 30 19:16:22.102249 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:22.102215 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6796af55-3e88-4dec-9b8a-9232d8c10e16" path="/var/lib/kubelet/pods/6796af55-3e88-4dec-9b8a-9232d8c10e16/volumes" Oct 30 19:16:22.515033 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:22.514994 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc"] Oct 30 19:16:22.515479 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:22.515462 4531 cpu_manager.go:414] "containerMap: removing container" podUID="587afca3-fc5e-4ddc-b0d3-d49713d4257d" containerName="revoke-keys" Oct 30 19:16:22.515529 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:22.515482 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="587afca3-fc5e-4ddc-b0d3-d49713d4257d" containerName="revoke-keys" Oct 30 19:16:22.515529 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:22.515521 4531 cpu_manager.go:414] "containerMap: removing container" podUID="6796af55-3e88-4dec-9b8a-9232d8c10e16" containerName="main" Oct 30 19:16:22.515529 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:22.515526 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6796af55-3e88-4dec-9b8a-9232d8c10e16" containerName="main" Oct 30 19:16:22.515632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:22.515614 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6796af55-3e88-4dec-9b8a-9232d8c10e16" containerName="main" Oct 30 19:16:22.515632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:22.515625 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="587afca3-fc5e-4ddc-b0d3-d49713d4257d" containerName="revoke-keys" Oct 30 19:16:22.520244 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:22.520214 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc" Oct 30 19:16:22.572457 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:22.572421 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:16:22.572457 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:22.572434 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:16:22.621966 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:22.621922 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:16:22.621966 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:22.621966 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:16:22.624953 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:22.624918 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc"] Oct 30 19:16:22.651628 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:22.651591 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a6e9c0-b670-4b7a-9113-e911c6b95b5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc\" (UID: \"12a6e9c0-b670-4b7a-9113-e911c6b95b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc" Oct 30 19:16:22.651628 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:22.651632 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8trkr\" (UniqueName: \"kubernetes.io/projected/12a6e9c0-b670-4b7a-9113-e911c6b95b5b-kube-api-access-8trkr\") pod \"maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc\" (UID: \"12a6e9c0-b670-4b7a-9113-e911c6b95b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc" Oct 30 19:16:22.651915 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:22.651749 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a6e9c0-b670-4b7a-9113-e911c6b95b5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc\" (UID: \"12a6e9c0-b670-4b7a-9113-e911c6b95b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc" Oct 30 19:16:22.753436 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:22.753391 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a6e9c0-b670-4b7a-9113-e911c6b95b5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc\" (UID: \"12a6e9c0-b670-4b7a-9113-e911c6b95b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc" Oct 30 19:16:22.753644 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:22.753447 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8trkr\" (UniqueName: \"kubernetes.io/projected/12a6e9c0-b670-4b7a-9113-e911c6b95b5b-kube-api-access-8trkr\") pod \"maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc\" (UID: \"12a6e9c0-b670-4b7a-9113-e911c6b95b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc" Oct 30 19:16:22.753644 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:22.753488 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a6e9c0-b670-4b7a-9113-e911c6b95b5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc\" (UID: \"12a6e9c0-b670-4b7a-9113-e911c6b95b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc" Oct 30 19:16:22.754271 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:22.754207 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a6e9c0-b670-4b7a-9113-e911c6b95b5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc\" (UID: \"12a6e9c0-b670-4b7a-9113-e911c6b95b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc" Oct 30 19:16:22.756050 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:22.756028 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a6e9c0-b670-4b7a-9113-e911c6b95b5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc\" (UID: \"12a6e9c0-b670-4b7a-9113-e911c6b95b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc" Oct 30 19:16:22.814841 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:22.814765 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8trkr\" (UniqueName: \"kubernetes.io/projected/12a6e9c0-b670-4b7a-9113-e911c6b95b5b-kube-api-access-8trkr\") pod \"maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc\" (UID: \"12a6e9c0-b670-4b7a-9113-e911c6b95b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc" Oct 30 19:16:22.834805 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:22.834765 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc" Oct 30 19:16:23.086831 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.086737 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07995457-g7p75"] Oct 30 19:16:23.091788 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.091758 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07995457-g7p75" Oct 30 19:16:23.149405 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.149359 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07995457-g7p75"] Oct 30 19:16:23.158253 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.158216 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5600e823-769f-4162-8d78-9855c0c5f24b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07995457-g7p75\" (UID: \"5600e823-769f-4162-8d78-9855c0c5f24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07995457-g7p75" Oct 30 19:16:23.158434 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.158386 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5600e823-769f-4162-8d78-9855c0c5f24b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07995457-g7p75\" (UID: \"5600e823-769f-4162-8d78-9855c0c5f24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07995457-g7p75" Oct 30 19:16:23.158480 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.158430 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sslg\" (UniqueName: \"kubernetes.io/projected/5600e823-769f-4162-8d78-9855c0c5f24b-kube-api-access-2sslg\") pod \"maas-api-revoke-sub-07995457-g7p75\" (UID: \"5600e823-769f-4162-8d78-9855c0c5f24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07995457-g7p75" Oct 30 19:16:23.259551 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.259506 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sslg\" (UniqueName: \"kubernetes.io/projected/5600e823-769f-4162-8d78-9855c0c5f24b-kube-api-access-2sslg\") pod \"maas-api-revoke-sub-07995457-g7p75\" (UID: \"5600e823-769f-4162-8d78-9855c0c5f24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07995457-g7p75" Oct 30 19:16:23.259765 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.259629 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5600e823-769f-4162-8d78-9855c0c5f24b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07995457-g7p75\" (UID: \"5600e823-769f-4162-8d78-9855c0c5f24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07995457-g7p75" Oct 30 19:16:23.259765 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.259724 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5600e823-769f-4162-8d78-9855c0c5f24b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07995457-g7p75\" (UID: \"5600e823-769f-4162-8d78-9855c0c5f24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07995457-g7p75" Oct 30 19:16:23.260831 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.260806 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5600e823-769f-4162-8d78-9855c0c5f24b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07995457-g7p75\" (UID: \"5600e823-769f-4162-8d78-9855c0c5f24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07995457-g7p75" Oct 30 19:16:23.261733 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.261702 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc"] Oct 30 19:16:23.262943 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.262911 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5600e823-769f-4162-8d78-9855c0c5f24b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07995457-g7p75\" (UID: \"5600e823-769f-4162-8d78-9855c0c5f24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07995457-g7p75" Oct 30 19:16:23.263426 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:16:23.263402 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a6e9c0_b670_4b7a_9113_e911c6b95b5b.slice/crio-7a30d7dabeb5c6f11ee3dbb55d4310d48504dcda24178ffe1a2478c01ce638c6 WatchSource:0}: Error finding container 7a30d7dabeb5c6f11ee3dbb55d4310d48504dcda24178ffe1a2478c01ce638c6: Status 404 returned error can't find the container with id 7a30d7dabeb5c6f11ee3dbb55d4310d48504dcda24178ffe1a2478c01ce638c6 Oct 30 19:16:23.320687 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.320657 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sslg\" (UniqueName: \"kubernetes.io/projected/5600e823-769f-4162-8d78-9855c0c5f24b-kube-api-access-2sslg\") pod \"maas-api-revoke-sub-07995457-g7p75\" (UID: \"5600e823-769f-4162-8d78-9855c0c5f24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07995457-g7p75" Oct 30 19:16:23.406903 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.406805 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07995457-g7p75" Oct 30 19:16:23.599454 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.599415 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4"] Oct 30 19:16:23.603494 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.603466 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.656010 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.655980 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-fd61e3-openshift-default-dockercfg-tn6nh\"" Oct 30 19:16:23.663791 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.663692 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-istio-data\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.663791 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.663738 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660d18a9-a78d-49f8-8e10-537b4a257731-istio-token\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.664020 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.663808 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-istio-envoy\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.664020 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.663960 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660d18a9-a78d-49f8-8e10-537b4a257731-istiod-ca-cert\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.664102 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.664058 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660d18a9-a78d-49f8-8e10-537b4a257731-istio-podinfo\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.664171 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.664117 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f67r\" (UniqueName: \"kubernetes.io/projected/660d18a9-a78d-49f8-8e10-537b4a257731-kube-api-access-9f67r\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.664224 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.664175 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-credential-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.664265 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.664235 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-workload-certs\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.664339 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.664262 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-workload-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.666457 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.666426 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4"] Oct 30 19:16:23.765630 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.765584 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660d18a9-a78d-49f8-8e10-537b4a257731-istio-podinfo\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.765805 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.765651 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f67r\" (UniqueName: \"kubernetes.io/projected/660d18a9-a78d-49f8-8e10-537b4a257731-kube-api-access-9f67r\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.765805 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.765678 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-credential-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.765805 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.765722 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-workload-certs\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.765805 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.765747 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-workload-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.766008 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.765816 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-istio-data\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.766008 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.765841 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660d18a9-a78d-49f8-8e10-537b4a257731-istio-token\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.766008 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.765877 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-istio-envoy\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.766008 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.765915 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660d18a9-a78d-49f8-8e10-537b4a257731-istiod-ca-cert\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.766542 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.766281 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-workload-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.766542 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.766310 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-workload-certs\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.766542 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.766502 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-istio-data\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.766782 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.766579 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-credential-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.766972 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.766881 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660d18a9-a78d-49f8-8e10-537b4a257731-istiod-ca-cert\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.768717 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.768688 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-istio-envoy\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.769046 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.769028 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660d18a9-a78d-49f8-8e10-537b4a257731-istio-podinfo\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.789895 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.789861 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07995457-g7p75"] Oct 30 19:16:23.792065 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:16:23.792026 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5600e823_769f_4162_8d78_9855c0c5f24b.slice/crio-70dff4e772c080119954ddaef0e9d134f217c01c8d1bdad8ec9c115e431f6c25 WatchSource:0}: Error finding container 70dff4e772c080119954ddaef0e9d134f217c01c8d1bdad8ec9c115e431f6c25: Status 404 returned error can't find the container with id 70dff4e772c080119954ddaef0e9d134f217c01c8d1bdad8ec9c115e431f6c25 Oct 30 19:16:23.827604 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.827572 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660d18a9-a78d-49f8-8e10-537b4a257731-istio-token\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.827750 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.827705 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f67r\" (UniqueName: \"kubernetes.io/projected/660d18a9-a78d-49f8-8e10-537b4a257731-kube-api-access-9f67r\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:23.880758 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.880717 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07995457-g7p75" event={"ID":"5600e823-769f-4162-8d78-9855c0c5f24b","Type":"ContainerStarted","Data":"70dff4e772c080119954ddaef0e9d134f217c01c8d1bdad8ec9c115e431f6c25"} Oct 30 19:16:23.882529 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.882489 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc" event={"ID":"12a6e9c0-b670-4b7a-9113-e911c6b95b5b","Type":"ContainerStarted","Data":"42a32c3c6cac17901ba64ec728b37f7c4e7d646888024e67edc8efd19f9eaab3"} Oct 30 19:16:23.882719 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.882553 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc" event={"ID":"12a6e9c0-b670-4b7a-9113-e911c6b95b5b","Type":"ContainerStarted","Data":"7a30d7dabeb5c6f11ee3dbb55d4310d48504dcda24178ffe1a2478c01ce638c6"} Oct 30 19:16:23.918221 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:23.918104 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:24.001145 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:24.001047 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc" podStartSLOduration=2.001025057 podStartE2EDuration="2.001025057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:23.998452836 +0000 UTC m=+2544.025557797" watchObservedRunningTime="2026-10-30 19:16:24.001025057 +0000 UTC m=+2544.028130017" Oct 30 19:16:24.305007 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:24.304978 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4"] Oct 30 19:16:24.306833 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:16:24.306796 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660d18a9_a78d_49f8_8e10_537b4a257731.slice/crio-59c9dab571b257f877313635faa76f2564f84b1534f2b5470e8c7bc515288960 WatchSource:0}: Error finding container 59c9dab571b257f877313635faa76f2564f84b1534f2b5470e8c7bc515288960: Status 404 returned error can't find the container with id 59c9dab571b257f877313635faa76f2564f84b1534f2b5470e8c7bc515288960 Oct 30 19:16:24.309567 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:24.309530 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:16:24.309654 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:24.309605 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:16:24.309654 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:24.309633 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:16:24.890716 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:24.890674 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" event={"ID":"660d18a9-a78d-49f8-8e10-537b4a257731","Type":"ContainerStarted","Data":"38c63315d96ea28e0551c2a21b38e602decdaf54bc2791faec85fc2e75d96b2d"} Oct 30 19:16:24.890887 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:24.890744 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" event={"ID":"660d18a9-a78d-49f8-8e10-537b4a257731","Type":"ContainerStarted","Data":"59c9dab571b257f877313635faa76f2564f84b1534f2b5470e8c7bc515288960"} Oct 30 19:16:24.892452 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:24.892423 4531 generic.go:356] "Generic (PLEG): container finished" podID="5600e823-769f-4162-8d78-9855c0c5f24b" containerID="78a85a3088eb2fa38b050c39da9e29d5a6598873a0c86b5d01233a044fec1356" exitCode=0 Oct 30 19:16:24.892605 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:24.892491 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07995457-g7p75" event={"ID":"5600e823-769f-4162-8d78-9855c0c5f24b","Type":"ContainerDied","Data":"78a85a3088eb2fa38b050c39da9e29d5a6598873a0c86b5d01233a044fec1356"} Oct 30 19:16:24.894330 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:24.894298 4531 generic.go:356] "Generic (PLEG): container finished" podID="12a6e9c0-b670-4b7a-9113-e911c6b95b5b" containerID="42a32c3c6cac17901ba64ec728b37f7c4e7d646888024e67edc8efd19f9eaab3" exitCode=0 Oct 30 19:16:24.894444 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:24.894386 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc" event={"ID":"12a6e9c0-b670-4b7a-9113-e911c6b95b5b","Type":"ContainerDied","Data":"42a32c3c6cac17901ba64ec728b37f7c4e7d646888024e67edc8efd19f9eaab3"} Oct 30 19:16:24.919362 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:24.919234 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:25.013238 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:25.013178 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" podStartSLOduration=2.013160069 podStartE2EDuration="2.013160069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:25.010916036 +0000 UTC m=+2545.038020995" watchObservedRunningTime="2026-10-30 19:16:25.013160069 +0000 UTC m=+2545.040265099" Oct 30 19:16:25.946605 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:25.946561 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 30 19:16:25.947185 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:25.946625 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" podUID="660d18a9-a78d-49f8-8e10-537b4a257731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 30 19:16:26.152066 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.151837 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc" Oct 30 19:16:26.178221 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.178186 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07995457-g7p75" Oct 30 19:16:26.313146 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.313096 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12a6e9c0-b670-4b7a-9113-e911c6b95b5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a6e9c0-b670-4b7a-9113-e911c6b95b5b-maas-api-xks-ca\") pod \"12a6e9c0-b670-4b7a-9113-e911c6b95b5b\" (UID: \"12a6e9c0-b670-4b7a-9113-e911c6b95b5b\") " Oct 30 19:16:26.313370 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.313179 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5600e823-769f-4162-8d78-9855c0c5f24b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5600e823-769f-4162-8d78-9855c0c5f24b-maas-api-openshift-ca\") pod \"5600e823-769f-4162-8d78-9855c0c5f24b\" (UID: \"5600e823-769f-4162-8d78-9855c0c5f24b\") " Oct 30 19:16:26.313370 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.313204 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5600e823-769f-4162-8d78-9855c0c5f24b-kube-api-access-2sslg\" (UniqueName: \"kubernetes.io/projected/5600e823-769f-4162-8d78-9855c0c5f24b-kube-api-access-2sslg\") pod \"5600e823-769f-4162-8d78-9855c0c5f24b\" (UID: \"5600e823-769f-4162-8d78-9855c0c5f24b\") " Oct 30 19:16:26.313370 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.313249 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5600e823-769f-4162-8d78-9855c0c5f24b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5600e823-769f-4162-8d78-9855c0c5f24b-maas-api-xks-ca\") pod \"5600e823-769f-4162-8d78-9855c0c5f24b\" (UID: \"5600e823-769f-4162-8d78-9855c0c5f24b\") " Oct 30 19:16:26.313370 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.313295 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12a6e9c0-b670-4b7a-9113-e911c6b95b5b-kube-api-access-8trkr\" (UniqueName: \"kubernetes.io/projected/12a6e9c0-b670-4b7a-9113-e911c6b95b5b-kube-api-access-8trkr\") pod \"12a6e9c0-b670-4b7a-9113-e911c6b95b5b\" (UID: \"12a6e9c0-b670-4b7a-9113-e911c6b95b5b\") " Oct 30 19:16:26.313740 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.313383 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12a6e9c0-b670-4b7a-9113-e911c6b95b5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a6e9c0-b670-4b7a-9113-e911c6b95b5b-maas-api-openshift-ca\") pod \"12a6e9c0-b670-4b7a-9113-e911c6b95b5b\" (UID: \"12a6e9c0-b670-4b7a-9113-e911c6b95b5b\") " Oct 30 19:16:26.313740 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.313649 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5600e823-769f-4162-8d78-9855c0c5f24b-maas-api-openshift-ca" pod "5600e823-769f-4162-8d78-9855c0c5f24b" (UID: "5600e823-769f-4162-8d78-9855c0c5f24b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:26.313881 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.313864 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5600e823-769f-4162-8d78-9855c0c5f24b-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:16:26.315009 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.313901 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12a6e9c0-b670-4b7a-9113-e911c6b95b5b-maas-api-openshift-ca" pod "12a6e9c0-b670-4b7a-9113-e911c6b95b5b" (UID: "12a6e9c0-b670-4b7a-9113-e911c6b95b5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:26.339408 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.335750 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a6e9c0-b670-4b7a-9113-e911c6b95b5b-kube-api-access-8trkr" pod "12a6e9c0-b670-4b7a-9113-e911c6b95b5b" (UID: "12a6e9c0-b670-4b7a-9113-e911c6b95b5b"). InnerVolumeSpecName "kube-api-access-8trkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:26.339408 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.336247 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5600e823-769f-4162-8d78-9855c0c5f24b-maas-api-xks-ca" pod "5600e823-769f-4162-8d78-9855c0c5f24b" (UID: "5600e823-769f-4162-8d78-9855c0c5f24b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:16:26.346351 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.346301 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12a6e9c0-b670-4b7a-9113-e911c6b95b5b-maas-api-xks-ca" pod "12a6e9c0-b670-4b7a-9113-e911c6b95b5b" (UID: "12a6e9c0-b670-4b7a-9113-e911c6b95b5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:16:26.346621 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.346595 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5600e823-769f-4162-8d78-9855c0c5f24b-kube-api-access-2sslg" pod "5600e823-769f-4162-8d78-9855c0c5f24b" (UID: "5600e823-769f-4162-8d78-9855c0c5f24b"). InnerVolumeSpecName "kube-api-access-2sslg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:26.406235 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:16:26.406115 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:16:26.428166 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.415224 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sslg\" (UniqueName: \"kubernetes.io/projected/5600e823-769f-4162-8d78-9855c0c5f24b-kube-api-access-2sslg\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:16:26.428166 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.415259 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5600e823-769f-4162-8d78-9855c0c5f24b-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:16:26.428166 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.415274 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8trkr\" (UniqueName: \"kubernetes.io/projected/12a6e9c0-b670-4b7a-9113-e911c6b95b5b-kube-api-access-8trkr\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:16:26.428166 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.415288 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a6e9c0-b670-4b7a-9113-e911c6b95b5b-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:16:26.428166 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.415301 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a6e9c0-b670-4b7a-9113-e911c6b95b5b-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:16:26.909144 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.909093 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07995457-g7p75" event={"ID":"5600e823-769f-4162-8d78-9855c0c5f24b","Type":"ContainerDied","Data":"70dff4e772c080119954ddaef0e9d134f217c01c8d1bdad8ec9c115e431f6c25"} Oct 30 19:16:26.909338 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.909104 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07995457-g7p75" Oct 30 19:16:26.909338 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.909212 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70dff4e772c080119954ddaef0e9d134f217c01c8d1bdad8ec9c115e431f6c25" Oct 30 19:16:26.910658 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.910630 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc" Oct 30 19:16:26.910775 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.910660 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc" event={"ID":"12a6e9c0-b670-4b7a-9113-e911c6b95b5b","Type":"ContainerDied","Data":"7a30d7dabeb5c6f11ee3dbb55d4310d48504dcda24178ffe1a2478c01ce638c6"} Oct 30 19:16:26.910775 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.910706 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a30d7dabeb5c6f11ee3dbb55d4310d48504dcda24178ffe1a2478c01ce638c6" Oct 30 19:16:26.928158 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.925787 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 30 19:16:26.928158 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:26.925871 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" podUID="660d18a9-a78d-49f8-8e10-537b4a257731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 30 19:16:27.936155 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:27.933639 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 30 19:16:27.936155 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:27.933725 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" podUID="660d18a9-a78d-49f8-8e10-537b4a257731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 30 19:16:28.923158 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:28.919590 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 30 19:16:28.923158 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:28.919655 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" podUID="660d18a9-a78d-49f8-8e10-537b4a257731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 30 19:16:29.926157 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:29.924605 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 30 19:16:29.926157 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:29.924668 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" podUID="660d18a9-a78d-49f8-8e10-537b4a257731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 30 19:16:30.920513 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:30.920475 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 30 19:16:30.920804 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:30.920763 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" podUID="660d18a9-a78d-49f8-8e10-537b4a257731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 30 19:16:31.923622 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:31.923587 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:31.924037 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:31.923904 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:31.924639 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:31.924617 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:16:33.099200 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:33.099156 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw"] Oct 30 19:16:35.508372 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:35.508334 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s"] Oct 30 19:16:35.556959 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:35.556923 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf43e7-84g6s"] Oct 30 19:16:36.098466 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:36.098428 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9be28974-d9c4-4009-b515-e9ed39b69772" path="/var/lib/kubelet/pods/9be28974-d9c4-4009-b515-e9ed39b69772/volumes" Oct 30 19:16:38.099830 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:38.099793 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw" podUID="84d03a78-d49d-45ee-a7f4-cc38e6e651cd" containerName="payload-processing" containerID="cri-o://993bd5d58ddd2d94d4251f46dd6515318b84632628cddd3a129b496e510afb19" gracePeriod=25 Oct 30 19:16:38.356213 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:38.356151 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw" Oct 30 19:16:38.465442 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:38.465410 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84d03a78-d49d-45ee-a7f4-cc38e6e651cd-kube-api-access-cxsth\" (UniqueName: \"kubernetes.io/projected/84d03a78-d49d-45ee-a7f4-cc38e6e651cd-kube-api-access-cxsth\") pod \"84d03a78-d49d-45ee-a7f4-cc38e6e651cd\" (UID: \"84d03a78-d49d-45ee-a7f4-cc38e6e651cd\") " Oct 30 19:16:38.465610 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:38.465523 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84d03a78-d49d-45ee-a7f4-cc38e6e651cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d03a78-d49d-45ee-a7f4-cc38e6e651cd-plugins-config-volume\") pod \"84d03a78-d49d-45ee-a7f4-cc38e6e651cd\" (UID: \"84d03a78-d49d-45ee-a7f4-cc38e6e651cd\") " Oct 30 19:16:38.465940 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:38.465911 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84d03a78-d49d-45ee-a7f4-cc38e6e651cd-plugins-config-volume" pod "84d03a78-d49d-45ee-a7f4-cc38e6e651cd" (UID: "84d03a78-d49d-45ee-a7f4-cc38e6e651cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:38.467640 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:38.467611 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d03a78-d49d-45ee-a7f4-cc38e6e651cd-kube-api-access-cxsth" pod "84d03a78-d49d-45ee-a7f4-cc38e6e651cd" (UID: "84d03a78-d49d-45ee-a7f4-cc38e6e651cd"). InnerVolumeSpecName "kube-api-access-cxsth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:38.566593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:38.566564 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxsth\" (UniqueName: \"kubernetes.io/projected/84d03a78-d49d-45ee-a7f4-cc38e6e651cd-kube-api-access-cxsth\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:16:38.566593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:38.566589 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84d03a78-d49d-45ee-a7f4-cc38e6e651cd-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:16:39.013439 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.013403 4531 generic.go:356] "Generic (PLEG): container finished" podID="84d03a78-d49d-45ee-a7f4-cc38e6e651cd" containerID="993bd5d58ddd2d94d4251f46dd6515318b84632628cddd3a129b496e510afb19" exitCode=0 Oct 30 19:16:39.013652 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.013454 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw" Oct 30 19:16:39.013652 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.013489 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw" event={"ID":"84d03a78-d49d-45ee-a7f4-cc38e6e651cd","Type":"ContainerDied","Data":"993bd5d58ddd2d94d4251f46dd6515318b84632628cddd3a129b496e510afb19"} Oct 30 19:16:39.013652 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.013548 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw" event={"ID":"84d03a78-d49d-45ee-a7f4-cc38e6e651cd","Type":"ContainerDied","Data":"b0f9241660d0ea0c095a36f6c9bf13a1cd01a3443dd2b78b71619a50a6e8c46a"} Oct 30 19:16:39.013652 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.013571 4531 scope.go:122] "RemoveContainer" containerID="993bd5d58ddd2d94d4251f46dd6515318b84632628cddd3a129b496e510afb19" Oct 30 19:16:39.023506 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.023490 4531 scope.go:122] "RemoveContainer" containerID="993bd5d58ddd2d94d4251f46dd6515318b84632628cddd3a129b496e510afb19" Oct 30 19:16:39.023778 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:16:39.023759 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"993bd5d58ddd2d94d4251f46dd6515318b84632628cddd3a129b496e510afb19\": container with ID starting with 993bd5d58ddd2d94d4251f46dd6515318b84632628cddd3a129b496e510afb19 not found: ID does not exist" containerID="993bd5d58ddd2d94d4251f46dd6515318b84632628cddd3a129b496e510afb19" Oct 30 19:16:39.023828 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.023789 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"993bd5d58ddd2d94d4251f46dd6515318b84632628cddd3a129b496e510afb19"} err="failed to get container status \"993bd5d58ddd2d94d4251f46dd6515318b84632628cddd3a129b496e510afb19\": rpc error: code = NotFound desc = could not find container \"993bd5d58ddd2d94d4251f46dd6515318b84632628cddd3a129b496e510afb19\": container with ID starting with 993bd5d58ddd2d94d4251f46dd6515318b84632628cddd3a129b496e510afb19 not found: ID does not exist" Oct 30 19:16:39.177919 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.177886 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw"] Oct 30 19:16:39.227508 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.227475 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f554bb-c94767df6-6twcw"] Oct 30 19:16:39.294891 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.294807 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13672578-mw2ns"] Oct 30 19:16:39.295266 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.295251 4531 cpu_manager.go:414] "containerMap: removing container" podUID="12a6e9c0-b670-4b7a-9113-e911c6b95b5b" containerName="revoke-keys" Oct 30 19:16:39.295353 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.295267 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12a6e9c0-b670-4b7a-9113-e911c6b95b5b" containerName="revoke-keys" Oct 30 19:16:39.295353 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.295284 4531 cpu_manager.go:414] "containerMap: removing container" podUID="5600e823-769f-4162-8d78-9855c0c5f24b" containerName="revoke-keys" Oct 30 19:16:39.295353 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.295289 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5600e823-769f-4162-8d78-9855c0c5f24b" containerName="revoke-keys" Oct 30 19:16:39.295353 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.295316 4531 cpu_manager.go:414] "containerMap: removing container" podUID="84d03a78-d49d-45ee-a7f4-cc38e6e651cd" containerName="payload-processing" Oct 30 19:16:39.295353 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.295323 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84d03a78-d49d-45ee-a7f4-cc38e6e651cd" containerName="payload-processing" Oct 30 19:16:39.295554 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.295389 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="84d03a78-d49d-45ee-a7f4-cc38e6e651cd" containerName="payload-processing" Oct 30 19:16:39.295554 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.295399 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="12a6e9c0-b670-4b7a-9113-e911c6b95b5b" containerName="revoke-keys" Oct 30 19:16:39.295554 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.295407 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5600e823-769f-4162-8d78-9855c0c5f24b" containerName="revoke-keys" Oct 30 19:16:39.299592 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.299568 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13672578-mw2ns" Oct 30 19:16:39.356460 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.356434 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:16:39.356655 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.356479 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:16:39.357252 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.357230 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:16:39.357450 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.357275 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:16:39.358965 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.358944 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13672578-mw2ns"] Oct 30 19:16:39.374966 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.374933 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bf6f231-31d4-4dbc-a6ac-64d55521f6b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13672578-mw2ns\" (UID: \"6bf6f231-31d4-4dbc-a6ac-64d55521f6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13672578-mw2ns" Oct 30 19:16:39.375219 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.375198 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bf6f231-31d4-4dbc-a6ac-64d55521f6b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13672578-mw2ns\" (UID: \"6bf6f231-31d4-4dbc-a6ac-64d55521f6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13672578-mw2ns" Oct 30 19:16:39.375314 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.375237 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr4bs\" (UniqueName: \"kubernetes.io/projected/6bf6f231-31d4-4dbc-a6ac-64d55521f6b3-kube-api-access-cr4bs\") pod \"maas-api-revoke-sub-13672578-mw2ns\" (UID: \"6bf6f231-31d4-4dbc-a6ac-64d55521f6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13672578-mw2ns" Oct 30 19:16:39.476184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.476121 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bf6f231-31d4-4dbc-a6ac-64d55521f6b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13672578-mw2ns\" (UID: \"6bf6f231-31d4-4dbc-a6ac-64d55521f6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13672578-mw2ns" Oct 30 19:16:39.476362 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.476194 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr4bs\" (UniqueName: \"kubernetes.io/projected/6bf6f231-31d4-4dbc-a6ac-64d55521f6b3-kube-api-access-cr4bs\") pod \"maas-api-revoke-sub-13672578-mw2ns\" (UID: \"6bf6f231-31d4-4dbc-a6ac-64d55521f6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13672578-mw2ns" Oct 30 19:16:39.476362 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.476228 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bf6f231-31d4-4dbc-a6ac-64d55521f6b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13672578-mw2ns\" (UID: \"6bf6f231-31d4-4dbc-a6ac-64d55521f6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13672578-mw2ns" Oct 30 19:16:39.476793 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.476764 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bf6f231-31d4-4dbc-a6ac-64d55521f6b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13672578-mw2ns\" (UID: \"6bf6f231-31d4-4dbc-a6ac-64d55521f6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13672578-mw2ns" Oct 30 19:16:39.478782 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.478762 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bf6f231-31d4-4dbc-a6ac-64d55521f6b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13672578-mw2ns\" (UID: \"6bf6f231-31d4-4dbc-a6ac-64d55521f6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13672578-mw2ns" Oct 30 19:16:39.537907 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.537867 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr4bs\" (UniqueName: \"kubernetes.io/projected/6bf6f231-31d4-4dbc-a6ac-64d55521f6b3-kube-api-access-cr4bs\") pod \"maas-api-revoke-sub-13672578-mw2ns\" (UID: \"6bf6f231-31d4-4dbc-a6ac-64d55521f6b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13672578-mw2ns" Oct 30 19:16:39.612159 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.612060 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13672578-mw2ns" Oct 30 19:16:39.988143 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:39.988105 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13672578-mw2ns"] Oct 30 19:16:39.989945 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:16:39.989908 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bf6f231_31d4_4dbc_a6ac_64d55521f6b3.slice/crio-f1c58949f3b4e2d0c4468e78c163d34b97ddb2f6e92949b926f6594d9b96e7ee WatchSource:0}: Error finding container f1c58949f3b4e2d0c4468e78c163d34b97ddb2f6e92949b926f6594d9b96e7ee: Status 404 returned error can't find the container with id f1c58949f3b4e2d0c4468e78c163d34b97ddb2f6e92949b926f6594d9b96e7ee Oct 30 19:16:40.021763 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:40.021732 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13672578-mw2ns" event={"ID":"6bf6f231-31d4-4dbc-a6ac-64d55521f6b3","Type":"ContainerStarted","Data":"f1c58949f3b4e2d0c4468e78c163d34b97ddb2f6e92949b926f6594d9b96e7ee"} Oct 30 19:16:40.099737 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:40.099691 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84d03a78-d49d-45ee-a7f4-cc38e6e651cd" path="/var/lib/kubelet/pods/84d03a78-d49d-45ee-a7f4-cc38e6e651cd/volumes" Oct 30 19:16:41.029555 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:41.029477 4531 generic.go:356] "Generic (PLEG): container finished" podID="6bf6f231-31d4-4dbc-a6ac-64d55521f6b3" containerID="1e1f944f324f8fe93e5abac6f30b729a7e50c357ec288620cdba341890611259" exitCode=0 Oct 30 19:16:41.029957 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:41.029566 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13672578-mw2ns" event={"ID":"6bf6f231-31d4-4dbc-a6ac-64d55521f6b3","Type":"ContainerDied","Data":"1e1f944f324f8fe93e5abac6f30b729a7e50c357ec288620cdba341890611259"} Oct 30 19:16:42.189806 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:42.189770 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13672578-mw2ns" Oct 30 19:16:42.307487 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:42.307393 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bf6f231-31d4-4dbc-a6ac-64d55521f6b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bf6f231-31d4-4dbc-a6ac-64d55521f6b3-maas-api-xks-ca\") pod \"6bf6f231-31d4-4dbc-a6ac-64d55521f6b3\" (UID: \"6bf6f231-31d4-4dbc-a6ac-64d55521f6b3\") " Oct 30 19:16:42.307676 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:42.307493 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bf6f231-31d4-4dbc-a6ac-64d55521f6b3-kube-api-access-cr4bs\" (UniqueName: \"kubernetes.io/projected/6bf6f231-31d4-4dbc-a6ac-64d55521f6b3-kube-api-access-cr4bs\") pod \"6bf6f231-31d4-4dbc-a6ac-64d55521f6b3\" (UID: \"6bf6f231-31d4-4dbc-a6ac-64d55521f6b3\") " Oct 30 19:16:42.307676 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:42.307627 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bf6f231-31d4-4dbc-a6ac-64d55521f6b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bf6f231-31d4-4dbc-a6ac-64d55521f6b3-maas-api-openshift-ca\") pod \"6bf6f231-31d4-4dbc-a6ac-64d55521f6b3\" (UID: \"6bf6f231-31d4-4dbc-a6ac-64d55521f6b3\") " Oct 30 19:16:42.308208 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:42.308175 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bf6f231-31d4-4dbc-a6ac-64d55521f6b3-maas-api-openshift-ca" pod "6bf6f231-31d4-4dbc-a6ac-64d55521f6b3" (UID: "6bf6f231-31d4-4dbc-a6ac-64d55521f6b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:42.310568 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:42.310534 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bf6f231-31d4-4dbc-a6ac-64d55521f6b3-maas-api-xks-ca" pod "6bf6f231-31d4-4dbc-a6ac-64d55521f6b3" (UID: "6bf6f231-31d4-4dbc-a6ac-64d55521f6b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:16:42.310568 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:42.310558 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bf6f231-31d4-4dbc-a6ac-64d55521f6b3-kube-api-access-cr4bs" pod "6bf6f231-31d4-4dbc-a6ac-64d55521f6b3" (UID: "6bf6f231-31d4-4dbc-a6ac-64d55521f6b3"). InnerVolumeSpecName "kube-api-access-cr4bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:42.409035 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:42.408996 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bf6f231-31d4-4dbc-a6ac-64d55521f6b3-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:16:42.409035 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:42.409028 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bf6f231-31d4-4dbc-a6ac-64d55521f6b3-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:16:42.409035 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:42.409039 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr4bs\" (UniqueName: \"kubernetes.io/projected/6bf6f231-31d4-4dbc-a6ac-64d55521f6b3-kube-api-access-cr4bs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:16:43.045852 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:43.045816 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13672578-mw2ns" Oct 30 19:16:43.046048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:43.045844 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13672578-mw2ns" event={"ID":"6bf6f231-31d4-4dbc-a6ac-64d55521f6b3","Type":"ContainerDied","Data":"f1c58949f3b4e2d0c4468e78c163d34b97ddb2f6e92949b926f6594d9b96e7ee"} Oct 30 19:16:43.046048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:43.045895 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1c58949f3b4e2d0c4468e78c163d34b97ddb2f6e92949b926f6594d9b96e7ee" Oct 30 19:16:47.346809 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:47.346772 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21ba810d-tm8xh"] Oct 30 19:16:47.347246 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:47.347236 4531 cpu_manager.go:414] "containerMap: removing container" podUID="6bf6f231-31d4-4dbc-a6ac-64d55521f6b3" containerName="revoke-keys" Oct 30 19:16:47.347293 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:47.347249 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bf6f231-31d4-4dbc-a6ac-64d55521f6b3" containerName="revoke-keys" Oct 30 19:16:47.347341 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:47.347331 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bf6f231-31d4-4dbc-a6ac-64d55521f6b3" containerName="revoke-keys" Oct 30 19:16:47.350428 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:47.350409 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ba810d-tm8xh" Oct 30 19:16:47.403614 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:47.403571 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:16:47.403813 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:47.403691 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:16:47.453082 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:47.453042 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:16:47.453082 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:47.453047 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:16:47.455748 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:47.455724 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21ba810d-tm8xh"] Oct 30 19:16:47.463996 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:47.463967 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzlt9\" (UniqueName: \"kubernetes.io/projected/88eec190-49a8-4d52-ada9-9546abe31e3d-kube-api-access-fzlt9\") pod \"maas-api-revoke-sub-21ba810d-tm8xh\" (UID: \"88eec190-49a8-4d52-ada9-9546abe31e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ba810d-tm8xh" Oct 30 19:16:47.464115 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:47.464090 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88eec190-49a8-4d52-ada9-9546abe31e3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21ba810d-tm8xh\" (UID: \"88eec190-49a8-4d52-ada9-9546abe31e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ba810d-tm8xh" Oct 30 19:16:47.464254 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:47.464237 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88eec190-49a8-4d52-ada9-9546abe31e3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21ba810d-tm8xh\" (UID: \"88eec190-49a8-4d52-ada9-9546abe31e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ba810d-tm8xh" Oct 30 19:16:47.565027 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:47.564992 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88eec190-49a8-4d52-ada9-9546abe31e3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21ba810d-tm8xh\" (UID: \"88eec190-49a8-4d52-ada9-9546abe31e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ba810d-tm8xh" Oct 30 19:16:47.565264 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:47.565067 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzlt9\" (UniqueName: \"kubernetes.io/projected/88eec190-49a8-4d52-ada9-9546abe31e3d-kube-api-access-fzlt9\") pod \"maas-api-revoke-sub-21ba810d-tm8xh\" (UID: \"88eec190-49a8-4d52-ada9-9546abe31e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ba810d-tm8xh" Oct 30 19:16:47.565264 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:47.565201 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88eec190-49a8-4d52-ada9-9546abe31e3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21ba810d-tm8xh\" (UID: \"88eec190-49a8-4d52-ada9-9546abe31e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ba810d-tm8xh" Oct 30 19:16:47.565987 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:47.565959 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88eec190-49a8-4d52-ada9-9546abe31e3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21ba810d-tm8xh\" (UID: \"88eec190-49a8-4d52-ada9-9546abe31e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ba810d-tm8xh" Oct 30 19:16:47.567571 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:47.567544 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88eec190-49a8-4d52-ada9-9546abe31e3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21ba810d-tm8xh\" (UID: \"88eec190-49a8-4d52-ada9-9546abe31e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ba810d-tm8xh" Oct 30 19:16:47.626172 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:47.626081 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzlt9\" (UniqueName: \"kubernetes.io/projected/88eec190-49a8-4d52-ada9-9546abe31e3d-kube-api-access-fzlt9\") pod \"maas-api-revoke-sub-21ba810d-tm8xh\" (UID: \"88eec190-49a8-4d52-ada9-9546abe31e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ba810d-tm8xh" Oct 30 19:16:47.663639 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:47.663604 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ba810d-tm8xh" Oct 30 19:16:48.047971 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:48.047909 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21ba810d-tm8xh"] Oct 30 19:16:48.050519 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:16:48.050486 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88eec190_49a8_4d52_ada9_9546abe31e3d.slice/crio-69c7cded46b095db9909ca57cf2982bd0427e448a53f6f9ba62b483099aaaff5 WatchSource:0}: Error finding container 69c7cded46b095db9909ca57cf2982bd0427e448a53f6f9ba62b483099aaaff5: Status 404 returned error can't find the container with id 69c7cded46b095db9909ca57cf2982bd0427e448a53f6f9ba62b483099aaaff5 Oct 30 19:16:48.076116 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:48.076083 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ba810d-tm8xh" event={"ID":"88eec190-49a8-4d52-ada9-9546abe31e3d","Type":"ContainerStarted","Data":"69c7cded46b095db9909ca57cf2982bd0427e448a53f6f9ba62b483099aaaff5"} Oct 30 19:16:49.083208 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:49.083170 4531 generic.go:356] "Generic (PLEG): container finished" podID="88eec190-49a8-4d52-ada9-9546abe31e3d" containerID="eb27180946a5f28be49f5c4d9bbc369714c07c15ea8e3378ee5144d619060b9a" exitCode=0 Oct 30 19:16:49.083661 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:49.083259 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ba810d-tm8xh" event={"ID":"88eec190-49a8-4d52-ada9-9546abe31e3d","Type":"ContainerDied","Data":"eb27180946a5f28be49f5c4d9bbc369714c07c15ea8e3378ee5144d619060b9a"} Oct 30 19:16:50.255890 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:50.255843 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ba810d-tm8xh" Oct 30 19:16:50.397035 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:50.396939 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88eec190-49a8-4d52-ada9-9546abe31e3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88eec190-49a8-4d52-ada9-9546abe31e3d-maas-api-openshift-ca\") pod \"88eec190-49a8-4d52-ada9-9546abe31e3d\" (UID: \"88eec190-49a8-4d52-ada9-9546abe31e3d\") " Oct 30 19:16:50.397232 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:50.397091 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88eec190-49a8-4d52-ada9-9546abe31e3d-kube-api-access-fzlt9\" (UniqueName: \"kubernetes.io/projected/88eec190-49a8-4d52-ada9-9546abe31e3d-kube-api-access-fzlt9\") pod \"88eec190-49a8-4d52-ada9-9546abe31e3d\" (UID: \"88eec190-49a8-4d52-ada9-9546abe31e3d\") " Oct 30 19:16:50.397232 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:50.397199 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88eec190-49a8-4d52-ada9-9546abe31e3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88eec190-49a8-4d52-ada9-9546abe31e3d-maas-api-xks-ca\") pod \"88eec190-49a8-4d52-ada9-9546abe31e3d\" (UID: \"88eec190-49a8-4d52-ada9-9546abe31e3d\") " Oct 30 19:16:50.397422 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:50.397397 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88eec190-49a8-4d52-ada9-9546abe31e3d-maas-api-openshift-ca" pod "88eec190-49a8-4d52-ada9-9546abe31e3d" (UID: "88eec190-49a8-4d52-ada9-9546abe31e3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:50.397678 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:50.397660 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88eec190-49a8-4d52-ada9-9546abe31e3d-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:16:50.399991 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:50.399957 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88eec190-49a8-4d52-ada9-9546abe31e3d-maas-api-xks-ca" pod "88eec190-49a8-4d52-ada9-9546abe31e3d" (UID: "88eec190-49a8-4d52-ada9-9546abe31e3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:16:50.399991 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:50.399973 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88eec190-49a8-4d52-ada9-9546abe31e3d-kube-api-access-fzlt9" pod "88eec190-49a8-4d52-ada9-9546abe31e3d" (UID: "88eec190-49a8-4d52-ada9-9546abe31e3d"). InnerVolumeSpecName "kube-api-access-fzlt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:50.498391 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:50.498347 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzlt9\" (UniqueName: \"kubernetes.io/projected/88eec190-49a8-4d52-ada9-9546abe31e3d-kube-api-access-fzlt9\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:16:50.498391 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:50.498384 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88eec190-49a8-4d52-ada9-9546abe31e3d-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:16:50.779895 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:50.779863 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp"] Oct 30 19:16:50.780331 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:50.780318 4531 cpu_manager.go:414] "containerMap: removing container" podUID="88eec190-49a8-4d52-ada9-9546abe31e3d" containerName="revoke-keys" Oct 30 19:16:50.780385 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:50.780333 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88eec190-49a8-4d52-ada9-9546abe31e3d" containerName="revoke-keys" Oct 30 19:16:50.780428 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:50.780419 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="88eec190-49a8-4d52-ada9-9546abe31e3d" containerName="revoke-keys" Oct 30 19:16:50.783589 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:50.783568 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp" Oct 30 19:16:50.844437 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:50.844382 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp"] Oct 30 19:16:50.904064 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:50.904031 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eecc6d95-8f95-4a8d-9b84-5945d02cd42b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp\" (UID: \"eecc6d95-8f95-4a8d-9b84-5945d02cd42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp" Oct 30 19:16:50.904064 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:50.904068 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eecc6d95-8f95-4a8d-9b84-5945d02cd42b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp\" (UID: \"eecc6d95-8f95-4a8d-9b84-5945d02cd42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp" Oct 30 19:16:50.904306 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:50.904088 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9z7v\" (UniqueName: \"kubernetes.io/projected/eecc6d95-8f95-4a8d-9b84-5945d02cd42b-kube-api-access-w9z7v\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp\" (UID: \"eecc6d95-8f95-4a8d-9b84-5945d02cd42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp" Oct 30 19:16:51.005296 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:51.005261 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eecc6d95-8f95-4a8d-9b84-5945d02cd42b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp\" (UID: \"eecc6d95-8f95-4a8d-9b84-5945d02cd42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp" Oct 30 19:16:51.005502 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:51.005303 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eecc6d95-8f95-4a8d-9b84-5945d02cd42b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp\" (UID: \"eecc6d95-8f95-4a8d-9b84-5945d02cd42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp" Oct 30 19:16:51.005502 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:51.005336 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9z7v\" (UniqueName: \"kubernetes.io/projected/eecc6d95-8f95-4a8d-9b84-5945d02cd42b-kube-api-access-w9z7v\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp\" (UID: \"eecc6d95-8f95-4a8d-9b84-5945d02cd42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp" Oct 30 19:16:51.005893 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:51.005871 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eecc6d95-8f95-4a8d-9b84-5945d02cd42b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp\" (UID: \"eecc6d95-8f95-4a8d-9b84-5945d02cd42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp" Oct 30 19:16:51.007898 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:51.007875 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eecc6d95-8f95-4a8d-9b84-5945d02cd42b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp\" (UID: \"eecc6d95-8f95-4a8d-9b84-5945d02cd42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp" Oct 30 19:16:51.068183 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:51.068076 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9z7v\" (UniqueName: \"kubernetes.io/projected/eecc6d95-8f95-4a8d-9b84-5945d02cd42b-kube-api-access-w9z7v\") pod \"maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp\" (UID: \"eecc6d95-8f95-4a8d-9b84-5945d02cd42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp" Oct 30 19:16:51.096529 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:51.096500 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ba810d-tm8xh" Oct 30 19:16:51.096696 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:51.096500 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ba810d-tm8xh" event={"ID":"88eec190-49a8-4d52-ada9-9546abe31e3d","Type":"ContainerDied","Data":"69c7cded46b095db9909ca57cf2982bd0427e448a53f6f9ba62b483099aaaff5"} Oct 30 19:16:51.096696 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:51.096618 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69c7cded46b095db9909ca57cf2982bd0427e448a53f6f9ba62b483099aaaff5" Oct 30 19:16:51.098912 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:51.098892 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp" Oct 30 19:16:51.511627 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:51.511599 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp"] Oct 30 19:16:51.512212 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:16:51.512182 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeecc6d95_8f95_4a8d_9b84_5945d02cd42b.slice/crio-4ae72a4bc4f377cdcf09c5ace92a19e0106b7875079b9b5fbd50e7dfcaf50667 WatchSource:0}: Error finding container 4ae72a4bc4f377cdcf09c5ace92a19e0106b7875079b9b5fbd50e7dfcaf50667: Status 404 returned error can't find the container with id 4ae72a4bc4f377cdcf09c5ace92a19e0106b7875079b9b5fbd50e7dfcaf50667 Oct 30 19:16:52.103179 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:52.103059 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp" event={"ID":"eecc6d95-8f95-4a8d-9b84-5945d02cd42b","Type":"ContainerStarted","Data":"b5dcda2adef101447573a0d827f15e5edf4910cef6e69ea796b62023a1926d04"} Oct 30 19:16:52.103179 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:52.103111 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp" event={"ID":"eecc6d95-8f95-4a8d-9b84-5945d02cd42b","Type":"ContainerStarted","Data":"4ae72a4bc4f377cdcf09c5ace92a19e0106b7875079b9b5fbd50e7dfcaf50667"} Oct 30 19:16:52.224163 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:52.224085 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp" podStartSLOduration=2.224073197 podStartE2EDuration="2.224073197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:52.221956365 +0000 UTC m=+2572.249061324" watchObservedRunningTime="2026-10-30 19:16:52.224073197 +0000 UTC m=+2572.251178156" Oct 30 19:16:53.114493 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:53.114460 4531 generic.go:356] "Generic (PLEG): container finished" podID="eecc6d95-8f95-4a8d-9b84-5945d02cd42b" containerID="b5dcda2adef101447573a0d827f15e5edf4910cef6e69ea796b62023a1926d04" exitCode=0 Oct 30 19:16:53.114946 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:53.114526 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp" event={"ID":"eecc6d95-8f95-4a8d-9b84-5945d02cd42b","Type":"ContainerDied","Data":"b5dcda2adef101447573a0d827f15e5edf4910cef6e69ea796b62023a1926d04"} Oct 30 19:16:54.257209 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:54.257183 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp" Oct 30 19:16:54.342063 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:54.341974 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eecc6d95-8f95-4a8d-9b84-5945d02cd42b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eecc6d95-8f95-4a8d-9b84-5945d02cd42b-maas-api-xks-ca\") pod \"eecc6d95-8f95-4a8d-9b84-5945d02cd42b\" (UID: \"eecc6d95-8f95-4a8d-9b84-5945d02cd42b\") " Oct 30 19:16:54.342262 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:54.342077 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eecc6d95-8f95-4a8d-9b84-5945d02cd42b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eecc6d95-8f95-4a8d-9b84-5945d02cd42b-maas-api-openshift-ca\") pod \"eecc6d95-8f95-4a8d-9b84-5945d02cd42b\" (UID: \"eecc6d95-8f95-4a8d-9b84-5945d02cd42b\") " Oct 30 19:16:54.342262 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:54.342110 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eecc6d95-8f95-4a8d-9b84-5945d02cd42b-kube-api-access-w9z7v\" (UniqueName: \"kubernetes.io/projected/eecc6d95-8f95-4a8d-9b84-5945d02cd42b-kube-api-access-w9z7v\") pod \"eecc6d95-8f95-4a8d-9b84-5945d02cd42b\" (UID: \"eecc6d95-8f95-4a8d-9b84-5945d02cd42b\") " Oct 30 19:16:54.342425 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:54.342404 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eecc6d95-8f95-4a8d-9b84-5945d02cd42b-maas-api-openshift-ca" pod "eecc6d95-8f95-4a8d-9b84-5945d02cd42b" (UID: "eecc6d95-8f95-4a8d-9b84-5945d02cd42b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:54.344424 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:54.344398 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eecc6d95-8f95-4a8d-9b84-5945d02cd42b-maas-api-xks-ca" pod "eecc6d95-8f95-4a8d-9b84-5945d02cd42b" (UID: "eecc6d95-8f95-4a8d-9b84-5945d02cd42b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:16:54.344546 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:54.344490 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eecc6d95-8f95-4a8d-9b84-5945d02cd42b-kube-api-access-w9z7v" pod "eecc6d95-8f95-4a8d-9b84-5945d02cd42b" (UID: "eecc6d95-8f95-4a8d-9b84-5945d02cd42b"). InnerVolumeSpecName "kube-api-access-w9z7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:54.443819 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:54.443780 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eecc6d95-8f95-4a8d-9b84-5945d02cd42b-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:16:54.443819 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:54.443810 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eecc6d95-8f95-4a8d-9b84-5945d02cd42b-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:16:54.443819 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:54.443822 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9z7v\" (UniqueName: \"kubernetes.io/projected/eecc6d95-8f95-4a8d-9b84-5945d02cd42b-kube-api-access-w9z7v\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:16:55.129259 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:55.129225 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp" event={"ID":"eecc6d95-8f95-4a8d-9b84-5945d02cd42b","Type":"ContainerDied","Data":"4ae72a4bc4f377cdcf09c5ace92a19e0106b7875079b9b5fbd50e7dfcaf50667"} Oct 30 19:16:55.129444 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:55.129276 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ae72a4bc4f377cdcf09c5ace92a19e0106b7875079b9b5fbd50e7dfcaf50667" Oct 30 19:16:55.129444 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:55.129280 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp" Oct 30 19:16:56.397611 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:16:56.397577 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:16:57.109960 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:57.109907 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc"] Oct 30 19:16:57.158562 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:57.158517 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f554bb-bnqtc"] Oct 30 19:16:58.098624 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:16:58.098591 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12a6e9c0-b670-4b7a-9113-e911c6b95b5b" path="/var/lib/kubelet/pods/12a6e9c0-b670-4b7a-9113-e911c6b95b5b/volumes" Oct 30 19:17:01.711938 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:01.711902 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d3bff4b-dxzb7"] Oct 30 19:17:01.712577 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:01.712554 4531 cpu_manager.go:414] "containerMap: removing container" podUID="eecc6d95-8f95-4a8d-9b84-5945d02cd42b" containerName="revoke-keys" Oct 30 19:17:01.712728 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:01.712580 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eecc6d95-8f95-4a8d-9b84-5945d02cd42b" containerName="revoke-keys" Oct 30 19:17:01.712728 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:01.712713 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="eecc6d95-8f95-4a8d-9b84-5945d02cd42b" containerName="revoke-keys" Oct 30 19:17:01.716361 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:01.716339 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3bff4b-dxzb7" Oct 30 19:17:01.768844 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:01.768808 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:17:01.769023 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:01.768869 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:17:01.818603 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:01.818573 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:17:01.818603 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:01.818581 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:17:01.821409 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:01.821382 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d3bff4b-dxzb7"] Oct 30 19:17:01.825281 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:01.825255 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l84n\" (UniqueName: \"kubernetes.io/projected/d22db1a8-7854-4889-a0f6-d2303262d880-kube-api-access-7l84n\") pod \"maas-api-revoke-sub-0d3bff4b-dxzb7\" (UID: \"d22db1a8-7854-4889-a0f6-d2303262d880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3bff4b-dxzb7" Oct 30 19:17:01.825364 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:01.825293 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d22db1a8-7854-4889-a0f6-d2303262d880-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d3bff4b-dxzb7\" (UID: \"d22db1a8-7854-4889-a0f6-d2303262d880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3bff4b-dxzb7" Oct 30 19:17:01.825447 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:01.825423 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d22db1a8-7854-4889-a0f6-d2303262d880-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d3bff4b-dxzb7\" (UID: \"d22db1a8-7854-4889-a0f6-d2303262d880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3bff4b-dxzb7" Oct 30 19:17:01.927325 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:01.927288 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l84n\" (UniqueName: \"kubernetes.io/projected/d22db1a8-7854-4889-a0f6-d2303262d880-kube-api-access-7l84n\") pod \"maas-api-revoke-sub-0d3bff4b-dxzb7\" (UID: \"d22db1a8-7854-4889-a0f6-d2303262d880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3bff4b-dxzb7" Oct 30 19:17:01.927325 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:01.927329 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d22db1a8-7854-4889-a0f6-d2303262d880-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d3bff4b-dxzb7\" (UID: \"d22db1a8-7854-4889-a0f6-d2303262d880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3bff4b-dxzb7" Oct 30 19:17:01.927562 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:01.927386 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d22db1a8-7854-4889-a0f6-d2303262d880-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d3bff4b-dxzb7\" (UID: \"d22db1a8-7854-4889-a0f6-d2303262d880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3bff4b-dxzb7" Oct 30 19:17:01.928025 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:01.928001 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d22db1a8-7854-4889-a0f6-d2303262d880-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d3bff4b-dxzb7\" (UID: \"d22db1a8-7854-4889-a0f6-d2303262d880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3bff4b-dxzb7" Oct 30 19:17:01.929996 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:01.929969 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d22db1a8-7854-4889-a0f6-d2303262d880-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d3bff4b-dxzb7\" (UID: \"d22db1a8-7854-4889-a0f6-d2303262d880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3bff4b-dxzb7" Oct 30 19:17:01.972367 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:01.972282 4531 scope.go:122] "RemoveContainer" containerID="52f7a1e12879e5fe88a7f3cda027806a733d24c7ee89e7643a09858dcde40fe4" Oct 30 19:17:01.982080 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:01.982049 4531 scope.go:122] "RemoveContainer" containerID="c39aaa8435065d3c0ac481ddd426e557deec7c2cde9d3880ce348a918787ab87" Oct 30 19:17:01.989046 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:01.989010 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l84n\" (UniqueName: \"kubernetes.io/projected/d22db1a8-7854-4889-a0f6-d2303262d880-kube-api-access-7l84n\") pod \"maas-api-revoke-sub-0d3bff4b-dxzb7\" (UID: \"d22db1a8-7854-4889-a0f6-d2303262d880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3bff4b-dxzb7" Oct 30 19:17:02.030211 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:02.030179 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3bff4b-dxzb7" Oct 30 19:17:02.408109 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:02.408084 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d3bff4b-dxzb7"] Oct 30 19:17:02.410074 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:17:02.410045 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22db1a8_7854_4889_a0f6_d2303262d880.slice/crio-f9972f7b9401d0e62937cda5fab6f7d6dd2774417fb2886289365d38d1286924 WatchSource:0}: Error finding container f9972f7b9401d0e62937cda5fab6f7d6dd2774417fb2886289365d38d1286924: Status 404 returned error can't find the container with id f9972f7b9401d0e62937cda5fab6f7d6dd2774417fb2886289365d38d1286924 Oct 30 19:17:03.177920 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:03.177878 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3bff4b-dxzb7" event={"ID":"d22db1a8-7854-4889-a0f6-d2303262d880","Type":"ContainerStarted","Data":"6a88d01225918396c0c70666b6b738dcd0a25ed1ce7901bca3bcb37eb99143b2"} Oct 30 19:17:03.178353 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:03.177934 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3bff4b-dxzb7" event={"ID":"d22db1a8-7854-4889-a0f6-d2303262d880","Type":"ContainerStarted","Data":"f9972f7b9401d0e62937cda5fab6f7d6dd2774417fb2886289365d38d1286924"} Oct 30 19:17:03.296255 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:03.296203 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3bff4b-dxzb7" podStartSLOduration=2.296187543 podStartE2EDuration="2.296187543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:03.295304682 +0000 UTC m=+2583.322409642" watchObservedRunningTime="2026-10-30 19:17:03.296187543 +0000 UTC m=+2583.323292502" Oct 30 19:17:04.185002 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:04.184965 4531 generic.go:356] "Generic (PLEG): container finished" podID="d22db1a8-7854-4889-a0f6-d2303262d880" containerID="6a88d01225918396c0c70666b6b738dcd0a25ed1ce7901bca3bcb37eb99143b2" exitCode=0 Oct 30 19:17:04.185443 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:04.185060 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3bff4b-dxzb7" event={"ID":"d22db1a8-7854-4889-a0f6-d2303262d880","Type":"ContainerDied","Data":"6a88d01225918396c0c70666b6b738dcd0a25ed1ce7901bca3bcb37eb99143b2"} Oct 30 19:17:05.179038 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:05.179004 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g"] Oct 30 19:17:05.182906 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:05.182883 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g" Oct 30 19:17:05.242319 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:05.242278 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g"] Oct 30 19:17:05.366754 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:05.366721 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e225e735-5256-4078-9c8f-9bfbcde0308e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g\" (UID: \"e225e735-5256-4078-9c8f-9bfbcde0308e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g" Oct 30 19:17:05.366980 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:05.366952 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67zvj\" (UniqueName: \"kubernetes.io/projected/e225e735-5256-4078-9c8f-9bfbcde0308e-kube-api-access-67zvj\") pod \"maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g\" (UID: \"e225e735-5256-4078-9c8f-9bfbcde0308e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g" Oct 30 19:17:05.367244 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:05.367219 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e225e735-5256-4078-9c8f-9bfbcde0308e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g\" (UID: \"e225e735-5256-4078-9c8f-9bfbcde0308e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g" Oct 30 19:17:05.370942 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:05.370916 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3bff4b-dxzb7" Oct 30 19:17:05.468422 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:05.468330 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d22db1a8-7854-4889-a0f6-d2303262d880-kube-api-access-7l84n\" (UniqueName: \"kubernetes.io/projected/d22db1a8-7854-4889-a0f6-d2303262d880-kube-api-access-7l84n\") pod \"d22db1a8-7854-4889-a0f6-d2303262d880\" (UID: \"d22db1a8-7854-4889-a0f6-d2303262d880\") " Oct 30 19:17:05.468422 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:05.468383 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d22db1a8-7854-4889-a0f6-d2303262d880-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d22db1a8-7854-4889-a0f6-d2303262d880-maas-api-openshift-ca\") pod \"d22db1a8-7854-4889-a0f6-d2303262d880\" (UID: \"d22db1a8-7854-4889-a0f6-d2303262d880\") " Oct 30 19:17:05.468628 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:05.468481 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d22db1a8-7854-4889-a0f6-d2303262d880-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d22db1a8-7854-4889-a0f6-d2303262d880-maas-api-xks-ca\") pod \"d22db1a8-7854-4889-a0f6-d2303262d880\" (UID: \"d22db1a8-7854-4889-a0f6-d2303262d880\") " Oct 30 19:17:05.468628 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:05.468602 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67zvj\" (UniqueName: \"kubernetes.io/projected/e225e735-5256-4078-9c8f-9bfbcde0308e-kube-api-access-67zvj\") pod \"maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g\" (UID: \"e225e735-5256-4078-9c8f-9bfbcde0308e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g" Oct 30 19:17:05.468725 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:05.468635 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e225e735-5256-4078-9c8f-9bfbcde0308e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g\" (UID: \"e225e735-5256-4078-9c8f-9bfbcde0308e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g" Oct 30 19:17:05.468725 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:05.468685 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e225e735-5256-4078-9c8f-9bfbcde0308e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g\" (UID: \"e225e735-5256-4078-9c8f-9bfbcde0308e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g" Oct 30 19:17:05.468841 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:05.468780 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d22db1a8-7854-4889-a0f6-d2303262d880-maas-api-openshift-ca" pod "d22db1a8-7854-4889-a0f6-d2303262d880" (UID: "d22db1a8-7854-4889-a0f6-d2303262d880"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:05.469443 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:05.469420 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e225e735-5256-4078-9c8f-9bfbcde0308e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g\" (UID: \"e225e735-5256-4078-9c8f-9bfbcde0308e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g" Oct 30 19:17:05.471321 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:05.471291 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d22db1a8-7854-4889-a0f6-d2303262d880-maas-api-xks-ca" pod "d22db1a8-7854-4889-a0f6-d2303262d880" (UID: "d22db1a8-7854-4889-a0f6-d2303262d880"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:05.471321 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:05.471314 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22db1a8-7854-4889-a0f6-d2303262d880-kube-api-access-7l84n" pod "d22db1a8-7854-4889-a0f6-d2303262d880" (UID: "d22db1a8-7854-4889-a0f6-d2303262d880"). InnerVolumeSpecName "kube-api-access-7l84n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:05.471492 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:05.471320 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e225e735-5256-4078-9c8f-9bfbcde0308e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g\" (UID: \"e225e735-5256-4078-9c8f-9bfbcde0308e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g" Oct 30 19:17:05.530118 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:05.530074 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67zvj\" (UniqueName: \"kubernetes.io/projected/e225e735-5256-4078-9c8f-9bfbcde0308e-kube-api-access-67zvj\") pod \"maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g\" (UID: \"e225e735-5256-4078-9c8f-9bfbcde0308e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g" Oct 30 19:17:05.569573 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:05.569532 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l84n\" (UniqueName: \"kubernetes.io/projected/d22db1a8-7854-4889-a0f6-d2303262d880-kube-api-access-7l84n\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:05.569573 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:05.569566 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d22db1a8-7854-4889-a0f6-d2303262d880-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:05.569573 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:05.569576 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d22db1a8-7854-4889-a0f6-d2303262d880-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:05.798054 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:05.798009 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g" Oct 30 19:17:06.180493 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:06.180466 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g"] Oct 30 19:17:06.181838 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:17:06.181814 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode225e735_5256_4078_9c8f_9bfbcde0308e.slice/crio-18eb5618c6ee1e14ce269d8e32c8811696b7273ef7e543eb9bf3b8aeb0b3858f WatchSource:0}: Error finding container 18eb5618c6ee1e14ce269d8e32c8811696b7273ef7e543eb9bf3b8aeb0b3858f: Status 404 returned error can't find the container with id 18eb5618c6ee1e14ce269d8e32c8811696b7273ef7e543eb9bf3b8aeb0b3858f Oct 30 19:17:06.198555 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:06.198521 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g" event={"ID":"e225e735-5256-4078-9c8f-9bfbcde0308e","Type":"ContainerStarted","Data":"18eb5618c6ee1e14ce269d8e32c8811696b7273ef7e543eb9bf3b8aeb0b3858f"} Oct 30 19:17:06.199917 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:06.199891 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3bff4b-dxzb7" Oct 30 19:17:06.200010 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:06.199919 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3bff4b-dxzb7" event={"ID":"d22db1a8-7854-4889-a0f6-d2303262d880","Type":"ContainerDied","Data":"f9972f7b9401d0e62937cda5fab6f7d6dd2774417fb2886289365d38d1286924"} Oct 30 19:17:06.200010 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:06.199968 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9972f7b9401d0e62937cda5fab6f7d6dd2774417fb2886289365d38d1286924" Oct 30 19:17:06.867754 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:06.867709 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt"] Oct 30 19:17:06.868232 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:06.867972 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" podUID="8256a12e-d979-4a98-8200-0b3937d26dae" containerName="istio-proxy" containerID="cri-o://6d3f486feff032cf596c79d4491c6a85029f43cf3acffce51c5ad025dd650323" gracePeriod=30 Oct 30 19:17:07.214599 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:07.214552 4531 generic.go:356] "Generic (PLEG): container finished" podID="e225e735-5256-4078-9c8f-9bfbcde0308e" containerID="b833ab47d1fb8c29a198f16d945adebcd3c313ba7d3763e4c2eb4730ab399cc0" exitCode=0 Oct 30 19:17:07.214814 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:07.214628 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g" event={"ID":"e225e735-5256-4078-9c8f-9bfbcde0308e","Type":"ContainerDied","Data":"b833ab47d1fb8c29a198f16d945adebcd3c313ba7d3763e4c2eb4730ab399cc0"} Oct 30 19:17:07.980425 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:07.980390 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79495d21-zvcbg"] Oct 30 19:17:07.980828 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:07.980798 4531 cpu_manager.go:414] "containerMap: removing container" podUID="d22db1a8-7854-4889-a0f6-d2303262d880" containerName="revoke-keys" Oct 30 19:17:07.980828 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:07.980811 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d22db1a8-7854-4889-a0f6-d2303262d880" containerName="revoke-keys" Oct 30 19:17:07.980914 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:07.980903 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d22db1a8-7854-4889-a0f6-d2303262d880" containerName="revoke-keys" Oct 30 19:17:07.984306 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:07.984282 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79495d21-zvcbg" Oct 30 19:17:07.995562 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:07.995529 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7q7l\" (UniqueName: \"kubernetes.io/projected/3a38212e-aab1-4e15-abd6-23ed981aac8f-kube-api-access-m7q7l\") pod \"maas-api-revoke-sub-79495d21-zvcbg\" (UID: \"3a38212e-aab1-4e15-abd6-23ed981aac8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79495d21-zvcbg" Oct 30 19:17:07.995687 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:07.995569 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a38212e-aab1-4e15-abd6-23ed981aac8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79495d21-zvcbg\" (UID: \"3a38212e-aab1-4e15-abd6-23ed981aac8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79495d21-zvcbg" Oct 30 19:17:07.995687 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:07.995598 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a38212e-aab1-4e15-abd6-23ed981aac8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79495d21-zvcbg\" (UID: \"3a38212e-aab1-4e15-abd6-23ed981aac8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79495d21-zvcbg" Oct 30 19:17:08.043999 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:08.043967 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79495d21-zvcbg"] Oct 30 19:17:08.096500 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:08.096468 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7q7l\" (UniqueName: \"kubernetes.io/projected/3a38212e-aab1-4e15-abd6-23ed981aac8f-kube-api-access-m7q7l\") pod \"maas-api-revoke-sub-79495d21-zvcbg\" (UID: \"3a38212e-aab1-4e15-abd6-23ed981aac8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79495d21-zvcbg" Oct 30 19:17:08.096686 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:08.096506 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a38212e-aab1-4e15-abd6-23ed981aac8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79495d21-zvcbg\" (UID: \"3a38212e-aab1-4e15-abd6-23ed981aac8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79495d21-zvcbg" Oct 30 19:17:08.096686 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:08.096531 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a38212e-aab1-4e15-abd6-23ed981aac8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79495d21-zvcbg\" (UID: \"3a38212e-aab1-4e15-abd6-23ed981aac8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79495d21-zvcbg" Oct 30 19:17:08.097221 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:08.097193 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a38212e-aab1-4e15-abd6-23ed981aac8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79495d21-zvcbg\" (UID: \"3a38212e-aab1-4e15-abd6-23ed981aac8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79495d21-zvcbg" Oct 30 19:17:08.099011 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:08.098991 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a38212e-aab1-4e15-abd6-23ed981aac8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79495d21-zvcbg\" (UID: \"3a38212e-aab1-4e15-abd6-23ed981aac8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79495d21-zvcbg" Oct 30 19:17:08.157463 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:08.157436 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7q7l\" (UniqueName: \"kubernetes.io/projected/3a38212e-aab1-4e15-abd6-23ed981aac8f-kube-api-access-m7q7l\") pod \"maas-api-revoke-sub-79495d21-zvcbg\" (UID: \"3a38212e-aab1-4e15-abd6-23ed981aac8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79495d21-zvcbg" Oct 30 19:17:08.296580 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:08.296545 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79495d21-zvcbg" Oct 30 19:17:08.389846 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:08.389813 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g" Oct 30 19:17:08.398711 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:08.398680 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e225e735-5256-4078-9c8f-9bfbcde0308e-kube-api-access-67zvj\" (UniqueName: \"kubernetes.io/projected/e225e735-5256-4078-9c8f-9bfbcde0308e-kube-api-access-67zvj\") pod \"e225e735-5256-4078-9c8f-9bfbcde0308e\" (UID: \"e225e735-5256-4078-9c8f-9bfbcde0308e\") " Oct 30 19:17:08.398906 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:08.398773 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e225e735-5256-4078-9c8f-9bfbcde0308e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e225e735-5256-4078-9c8f-9bfbcde0308e-maas-api-openshift-ca\") pod \"e225e735-5256-4078-9c8f-9bfbcde0308e\" (UID: \"e225e735-5256-4078-9c8f-9bfbcde0308e\") " Oct 30 19:17:08.398906 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:08.398833 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e225e735-5256-4078-9c8f-9bfbcde0308e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e225e735-5256-4078-9c8f-9bfbcde0308e-maas-api-xks-ca\") pod \"e225e735-5256-4078-9c8f-9bfbcde0308e\" (UID: \"e225e735-5256-4078-9c8f-9bfbcde0308e\") " Oct 30 19:17:08.399220 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:08.399187 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e225e735-5256-4078-9c8f-9bfbcde0308e-maas-api-openshift-ca" pod "e225e735-5256-4078-9c8f-9bfbcde0308e" (UID: "e225e735-5256-4078-9c8f-9bfbcde0308e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:08.401405 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:08.401364 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e225e735-5256-4078-9c8f-9bfbcde0308e-maas-api-xks-ca" pod "e225e735-5256-4078-9c8f-9bfbcde0308e" (UID: "e225e735-5256-4078-9c8f-9bfbcde0308e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:08.401516 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:08.401404 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e225e735-5256-4078-9c8f-9bfbcde0308e-kube-api-access-67zvj" pod "e225e735-5256-4078-9c8f-9bfbcde0308e" (UID: "e225e735-5256-4078-9c8f-9bfbcde0308e"). InnerVolumeSpecName "kube-api-access-67zvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:08.500382 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:08.500317 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e225e735-5256-4078-9c8f-9bfbcde0308e-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.500382 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:08.500372 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67zvj\" (UniqueName: \"kubernetes.io/projected/e225e735-5256-4078-9c8f-9bfbcde0308e-kube-api-access-67zvj\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.500382 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:08.500386 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e225e735-5256-4078-9c8f-9bfbcde0308e-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:08.679167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:08.679095 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79495d21-zvcbg"] Oct 30 19:17:08.681619 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:17:08.681591 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a38212e_aab1_4e15_abd6_23ed981aac8f.slice/crio-ffd1821839ef4740614a822dee641743d1512845f514b1f729cc6852aa06312e WatchSource:0}: Error finding container ffd1821839ef4740614a822dee641743d1512845f514b1f729cc6852aa06312e: Status 404 returned error can't find the container with id ffd1821839ef4740614a822dee641743d1512845f514b1f729cc6852aa06312e Oct 30 19:17:08.970734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:08.970691 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb"] Oct 30 19:17:08.971231 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:08.971212 4531 cpu_manager.go:414] "containerMap: removing container" podUID="e225e735-5256-4078-9c8f-9bfbcde0308e" containerName="revoke-keys" Oct 30 19:17:08.971332 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:08.971233 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e225e735-5256-4078-9c8f-9bfbcde0308e" containerName="revoke-keys" Oct 30 19:17:08.971390 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:08.971348 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e225e735-5256-4078-9c8f-9bfbcde0308e" containerName="revoke-keys" Oct 30 19:17:08.974619 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:08.974594 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.007192 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.007152 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-data\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.007657 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.007251 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-podinfo\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.007657 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.007304 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-workload-certs\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.007657 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.007338 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-credential-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.007657 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.007398 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-token\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.007657 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.007427 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istiod-ca-cert\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.007657 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.007459 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-workload-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.007657 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.007549 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sw7n\" (UniqueName: \"kubernetes.io/projected/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-kube-api-access-9sw7n\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.007657 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.007639 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-envoy\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.080016 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.079981 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb"] Oct 30 19:17:09.108884 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.108842 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-podinfo\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.109079 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.108903 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-workload-certs\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.109079 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.108952 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-credential-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.109079 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.108985 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-token\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.109079 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.109006 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istiod-ca-cert\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.109079 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.109033 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-workload-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.109079 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.109065 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sw7n\" (UniqueName: \"kubernetes.io/projected/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-kube-api-access-9sw7n\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.109445 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.109156 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-envoy\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.109445 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.109236 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-data\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.109445 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.109331 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-workload-certs\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.109874 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.109589 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-workload-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.109874 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.109688 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-data\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.110347 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.110287 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istiod-ca-cert\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.110733 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.110703 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-credential-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.112120 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.112098 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-envoy\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.112386 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.112365 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-podinfo\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.171513 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.171479 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-token\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.176245 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.176206 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sw7n\" (UniqueName: \"kubernetes.io/projected/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-kube-api-access-9sw7n\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.234041 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.233918 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g" event={"ID":"e225e735-5256-4078-9c8f-9bfbcde0308e","Type":"ContainerDied","Data":"18eb5618c6ee1e14ce269d8e32c8811696b7273ef7e543eb9bf3b8aeb0b3858f"} Oct 30 19:17:09.234041 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.234000 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18eb5618c6ee1e14ce269d8e32c8811696b7273ef7e543eb9bf3b8aeb0b3858f" Oct 30 19:17:09.234041 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.233935 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g" Oct 30 19:17:09.236923 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.236062 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79495d21-zvcbg" event={"ID":"3a38212e-aab1-4e15-abd6-23ed981aac8f","Type":"ContainerStarted","Data":"fff0965ee53b9e07bf6eb60b16f719f5cc0c4a4bd2eada1b242ad8cc98029805"} Oct 30 19:17:09.236923 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.236171 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79495d21-zvcbg" event={"ID":"3a38212e-aab1-4e15-abd6-23ed981aac8f","Type":"ContainerStarted","Data":"ffd1821839ef4740614a822dee641743d1512845f514b1f729cc6852aa06312e"} Oct 30 19:17:09.287625 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.287585 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:09.354997 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.354927 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79495d21-zvcbg" podStartSLOduration=2.354873205 podStartE2EDuration="2.354873205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:09.353678338 +0000 UTC m=+2589.380783295" watchObservedRunningTime="2026-10-30 19:17:09.354873205 +0000 UTC m=+2589.381978166" Oct 30 19:17:09.718248 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.718217 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb"] Oct 30 19:17:09.720436 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:17:09.720407 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93ef69c_5501_4a9f_a9f8_9b3b1c8ab1e4.slice/crio-2da6b1f176901b28473f382f1f9d7b3fe4fee7ea291cbd0f214696e99454db23 WatchSource:0}: Error finding container 2da6b1f176901b28473f382f1f9d7b3fe4fee7ea291cbd0f214696e99454db23: Status 404 returned error can't find the container with id 2da6b1f176901b28473f382f1f9d7b3fe4fee7ea291cbd0f214696e99454db23 Oct 30 19:17:09.723153 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.723092 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:17:09.723250 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.723183 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:17:09.723250 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:09.723216 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:17:10.253206 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:10.253153 4531 generic.go:356] "Generic (PLEG): container finished" podID="3a38212e-aab1-4e15-abd6-23ed981aac8f" containerID="fff0965ee53b9e07bf6eb60b16f719f5cc0c4a4bd2eada1b242ad8cc98029805" exitCode=0 Oct 30 19:17:10.253674 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:10.253229 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79495d21-zvcbg" event={"ID":"3a38212e-aab1-4e15-abd6-23ed981aac8f","Type":"ContainerDied","Data":"fff0965ee53b9e07bf6eb60b16f719f5cc0c4a4bd2eada1b242ad8cc98029805"} Oct 30 19:17:10.255197 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:10.255170 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" event={"ID":"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4","Type":"ContainerStarted","Data":"7f2d5aae037dbdc4490458e36b63877d849c855fbc0c9f7f6087baeed5addb42"} Oct 30 19:17:10.255320 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:10.255215 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" event={"ID":"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4","Type":"ContainerStarted","Data":"2da6b1f176901b28473f382f1f9d7b3fe4fee7ea291cbd0f214696e99454db23"} Oct 30 19:17:10.288396 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:10.288358 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:10.290204 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:10.290171 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 30 19:17:10.290346 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:10.290236 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" podUID="f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 30 19:17:10.495847 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:10.495784 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" podStartSLOduration=2.495768068 podStartE2EDuration="2.495768068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:10.492012374 +0000 UTC m=+2590.519117338" watchObservedRunningTime="2026-10-30 19:17:10.495768068 +0000 UTC m=+2590.522873027" Oct 30 19:17:11.288320 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:11.288284 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 30 19:17:11.288726 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:11.288341 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" podUID="f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 30 19:17:11.452153 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:11.452097 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79495d21-zvcbg" Oct 30 19:17:11.562923 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:11.562821 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a38212e-aab1-4e15-abd6-23ed981aac8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a38212e-aab1-4e15-abd6-23ed981aac8f-maas-api-openshift-ca\") pod \"3a38212e-aab1-4e15-abd6-23ed981aac8f\" (UID: \"3a38212e-aab1-4e15-abd6-23ed981aac8f\") " Oct 30 19:17:11.564237 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:11.564196 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a38212e-aab1-4e15-abd6-23ed981aac8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a38212e-aab1-4e15-abd6-23ed981aac8f-maas-api-xks-ca\") pod \"3a38212e-aab1-4e15-abd6-23ed981aac8f\" (UID: \"3a38212e-aab1-4e15-abd6-23ed981aac8f\") " Oct 30 19:17:11.564389 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:11.564351 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a38212e-aab1-4e15-abd6-23ed981aac8f-kube-api-access-m7q7l\" (UniqueName: \"kubernetes.io/projected/3a38212e-aab1-4e15-abd6-23ed981aac8f-kube-api-access-m7q7l\") pod \"3a38212e-aab1-4e15-abd6-23ed981aac8f\" (UID: \"3a38212e-aab1-4e15-abd6-23ed981aac8f\") " Oct 30 19:17:11.564716 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:11.564692 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a38212e-aab1-4e15-abd6-23ed981aac8f-maas-api-openshift-ca" pod "3a38212e-aab1-4e15-abd6-23ed981aac8f" (UID: "3a38212e-aab1-4e15-abd6-23ed981aac8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:11.566059 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:11.566034 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a38212e-aab1-4e15-abd6-23ed981aac8f-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:11.568500 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:11.568381 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a38212e-aab1-4e15-abd6-23ed981aac8f-kube-api-access-m7q7l" pod "3a38212e-aab1-4e15-abd6-23ed981aac8f" (UID: "3a38212e-aab1-4e15-abd6-23ed981aac8f"). InnerVolumeSpecName "kube-api-access-m7q7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:11.568500 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:11.568412 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a38212e-aab1-4e15-abd6-23ed981aac8f-maas-api-xks-ca" pod "3a38212e-aab1-4e15-abd6-23ed981aac8f" (UID: "3a38212e-aab1-4e15-abd6-23ed981aac8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:11.666962 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:11.666923 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a38212e-aab1-4e15-abd6-23ed981aac8f-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:11.666962 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:11.666953 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7q7l\" (UniqueName: \"kubernetes.io/projected/3a38212e-aab1-4e15-abd6-23ed981aac8f-kube-api-access-m7q7l\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:12.149675 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.149649 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 19:17:12.172386 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.172354 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-credential-socket\") pod \"8256a12e-d979-4a98-8200-0b3937d26dae\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " Oct 30 19:17:12.172572 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.172397 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8256a12e-d979-4a98-8200-0b3937d26dae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8256a12e-d979-4a98-8200-0b3937d26dae-istiod-ca-cert\") pod \"8256a12e-d979-4a98-8200-0b3937d26dae\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " Oct 30 19:17:12.172572 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.172463 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8256a12e-d979-4a98-8200-0b3937d26dae-kube-api-access-h9mld\" (UniqueName: \"kubernetes.io/projected/8256a12e-d979-4a98-8200-0b3937d26dae-kube-api-access-h9mld\") pod \"8256a12e-d979-4a98-8200-0b3937d26dae\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " Oct 30 19:17:12.172572 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.172489 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8256a12e-d979-4a98-8200-0b3937d26dae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8256a12e-d979-4a98-8200-0b3937d26dae-istio-podinfo\") pod \"8256a12e-d979-4a98-8200-0b3937d26dae\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " Oct 30 19:17:12.172750 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.172581 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-workload-socket\") pod \"8256a12e-d979-4a98-8200-0b3937d26dae\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " Oct 30 19:17:12.172750 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.172633 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8256a12e-d979-4a98-8200-0b3937d26dae-istio-token\" (UniqueName: \"kubernetes.io/projected/8256a12e-d979-4a98-8200-0b3937d26dae-istio-token\") pod \"8256a12e-d979-4a98-8200-0b3937d26dae\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " Oct 30 19:17:12.172750 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.172646 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-credential-socket" pod "8256a12e-d979-4a98-8200-0b3937d26dae" (UID: "8256a12e-d979-4a98-8200-0b3937d26dae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:12.172750 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.172658 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-workload-certs\") pod \"8256a12e-d979-4a98-8200-0b3937d26dae\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " Oct 30 19:17:12.172977 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.172799 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-istio-envoy\") pod \"8256a12e-d979-4a98-8200-0b3937d26dae\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " Oct 30 19:17:12.172977 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.172827 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-istio-data\") pod \"8256a12e-d979-4a98-8200-0b3937d26dae\" (UID: \"8256a12e-d979-4a98-8200-0b3937d26dae\") " Oct 30 19:17:12.172977 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.172854 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-workload-certs" pod "8256a12e-d979-4a98-8200-0b3937d26dae" (UID: "8256a12e-d979-4a98-8200-0b3937d26dae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:12.172977 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.172925 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8256a12e-d979-4a98-8200-0b3937d26dae-istiod-ca-cert" pod "8256a12e-d979-4a98-8200-0b3937d26dae" (UID: "8256a12e-d979-4a98-8200-0b3937d26dae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:12.173230 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.173158 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-workload-socket" pod "8256a12e-d979-4a98-8200-0b3937d26dae" (UID: "8256a12e-d979-4a98-8200-0b3937d26dae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:12.173386 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.173308 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:12.173386 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.173330 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8256a12e-d979-4a98-8200-0b3937d26dae-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:12.173386 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.173343 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:12.173386 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.173359 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:12.173623 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.173442 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-istio-data" pod "8256a12e-d979-4a98-8200-0b3937d26dae" (UID: "8256a12e-d979-4a98-8200-0b3937d26dae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:12.174846 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.174818 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8256a12e-d979-4a98-8200-0b3937d26dae-kube-api-access-h9mld" pod "8256a12e-d979-4a98-8200-0b3937d26dae" (UID: "8256a12e-d979-4a98-8200-0b3937d26dae"). InnerVolumeSpecName "kube-api-access-h9mld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:12.175092 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.175071 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8256a12e-d979-4a98-8200-0b3937d26dae-istio-token" pod "8256a12e-d979-4a98-8200-0b3937d26dae" (UID: "8256a12e-d979-4a98-8200-0b3937d26dae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:12.175418 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.175395 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8256a12e-d979-4a98-8200-0b3937d26dae-istio-podinfo" pod "8256a12e-d979-4a98-8200-0b3937d26dae" (UID: "8256a12e-d979-4a98-8200-0b3937d26dae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:17:12.175508 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.175413 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-istio-envoy" pod "8256a12e-d979-4a98-8200-0b3937d26dae" (UID: "8256a12e-d979-4a98-8200-0b3937d26dae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:12.269472 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.269430 4531 generic.go:356] "Generic (PLEG): container finished" podID="8256a12e-d979-4a98-8200-0b3937d26dae" containerID="6d3f486feff032cf596c79d4491c6a85029f43cf3acffce51c5ad025dd650323" exitCode=0 Oct 30 19:17:12.269660 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.269503 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" Oct 30 19:17:12.269660 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.269524 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" event={"ID":"8256a12e-d979-4a98-8200-0b3937d26dae","Type":"ContainerDied","Data":"6d3f486feff032cf596c79d4491c6a85029f43cf3acffce51c5ad025dd650323"} Oct 30 19:17:12.269660 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.269604 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt" event={"ID":"8256a12e-d979-4a98-8200-0b3937d26dae","Type":"ContainerDied","Data":"ddcd7319168278436b71ebab5af221bb96795c69254c3fd45a732b35f349a4dc"} Oct 30 19:17:12.269660 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.269652 4531 scope.go:122] "RemoveContainer" containerID="6d3f486feff032cf596c79d4491c6a85029f43cf3acffce51c5ad025dd650323" Oct 30 19:17:12.274528 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.273277 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79495d21-zvcbg" Oct 30 19:17:12.274528 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.274363 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79495d21-zvcbg" event={"ID":"3a38212e-aab1-4e15-abd6-23ed981aac8f","Type":"ContainerDied","Data":"ffd1821839ef4740614a822dee641743d1512845f514b1f729cc6852aa06312e"} Oct 30 19:17:12.274528 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.274408 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffd1821839ef4740614a822dee641743d1512845f514b1f729cc6852aa06312e" Oct 30 19:17:12.275192 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.275103 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9mld\" (UniqueName: \"kubernetes.io/projected/8256a12e-d979-4a98-8200-0b3937d26dae-kube-api-access-h9mld\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:12.275192 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.275153 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8256a12e-d979-4a98-8200-0b3937d26dae-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:12.275192 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.275169 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8256a12e-d979-4a98-8200-0b3937d26dae-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:12.275192 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.275185 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:12.275460 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.275200 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8256a12e-d979-4a98-8200-0b3937d26dae-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:12.283917 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.283896 4531 scope.go:122] "RemoveContainer" containerID="6d3f486feff032cf596c79d4491c6a85029f43cf3acffce51c5ad025dd650323" Oct 30 19:17:12.284284 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:17:12.284266 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d3f486feff032cf596c79d4491c6a85029f43cf3acffce51c5ad025dd650323\": container with ID starting with 6d3f486feff032cf596c79d4491c6a85029f43cf3acffce51c5ad025dd650323 not found: ID does not exist" containerID="6d3f486feff032cf596c79d4491c6a85029f43cf3acffce51c5ad025dd650323" Oct 30 19:17:12.284360 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.284292 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d3f486feff032cf596c79d4491c6a85029f43cf3acffce51c5ad025dd650323"} err="failed to get container status \"6d3f486feff032cf596c79d4491c6a85029f43cf3acffce51c5ad025dd650323\": rpc error: code = NotFound desc = could not find container \"6d3f486feff032cf596c79d4491c6a85029f43cf3acffce51c5ad025dd650323\": container with ID starting with 6d3f486feff032cf596c79d4491c6a85029f43cf3acffce51c5ad025dd650323 not found: ID does not exist" Oct 30 19:17:12.288642 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.288617 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 30 19:17:12.289025 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.288672 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" podUID="f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 30 19:17:12.454008 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.453971 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt"] Oct 30 19:17:12.459348 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:12.459316 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-66db998887-nkqzt"] Oct 30 19:17:13.288054 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:13.288018 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 30 19:17:13.288279 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:13.288072 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" podUID="f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 30 19:17:13.944154 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:13.943739 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62"] Oct 30 19:17:13.944154 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:13.943989 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62" podUID="5bbecb8e-40ce-4b0d-948f-c7b53598da3f" containerName="payload-processing" containerID="cri-o://5298e821c3f4ff6dc1e9ac0d9e8830833ac86291d9468266d2eaf04340b22ad1" gracePeriod=30 Oct 30 19:17:13.956309 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:13.956277 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62" podUID="5bbecb8e-40ce-4b0d-948f-c7b53598da3f" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.162:9004: connect: connection refused" Oct 30 19:17:13.992255 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:13.992223 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs"] Oct 30 19:17:13.992429 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:13.992410 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs" podUID="0677eeba-3e54-4b81-a294-38435cc77ef9" containerName="payload-pre-processing" containerID="cri-o://dcbd49ddbc3af98ba7ea895af733ce704522d2d7d988e31bece622b0bf6a7e6c" gracePeriod=30 Oct 30 19:17:14.146195 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:14.140260 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8256a12e-d979-4a98-8200-0b3937d26dae" path="/var/lib/kubelet/pods/8256a12e-d979-4a98-8200-0b3937d26dae/volumes" Oct 30 19:17:14.288498 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:14.288066 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 30 19:17:14.288498 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:14.288169 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" podUID="f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 30 19:17:14.289383 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:14.289349 4531 generic.go:356] "Generic (PLEG): container finished" podID="5bbecb8e-40ce-4b0d-948f-c7b53598da3f" containerID="5298e821c3f4ff6dc1e9ac0d9e8830833ac86291d9468266d2eaf04340b22ad1" exitCode=0 Oct 30 19:17:14.289540 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:14.289395 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62" event={"ID":"5bbecb8e-40ce-4b0d-948f-c7b53598da3f","Type":"ContainerDied","Data":"5298e821c3f4ff6dc1e9ac0d9e8830833ac86291d9468266d2eaf04340b22ad1"} Oct 30 19:17:14.291211 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:14.291186 4531 generic.go:356] "Generic (PLEG): container finished" podID="0677eeba-3e54-4b81-a294-38435cc77ef9" containerID="dcbd49ddbc3af98ba7ea895af733ce704522d2d7d988e31bece622b0bf6a7e6c" exitCode=0 Oct 30 19:17:14.291427 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:14.291244 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs" event={"ID":"0677eeba-3e54-4b81-a294-38435cc77ef9","Type":"ContainerDied","Data":"dcbd49ddbc3af98ba7ea895af733ce704522d2d7d988e31bece622b0bf6a7e6c"} Oct 30 19:17:14.355688 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:14.355662 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62" Oct 30 19:17:14.360487 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:14.360464 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs" Oct 30 19:17:14.402618 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:14.402590 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0677eeba-3e54-4b81-a294-38435cc77ef9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0677eeba-3e54-4b81-a294-38435cc77ef9-plugins-config-volume\") pod \"0677eeba-3e54-4b81-a294-38435cc77ef9\" (UID: \"0677eeba-3e54-4b81-a294-38435cc77ef9\") " Oct 30 19:17:14.402724 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:14.402648 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0677eeba-3e54-4b81-a294-38435cc77ef9-kube-api-access-rm844\" (UniqueName: \"kubernetes.io/projected/0677eeba-3e54-4b81-a294-38435cc77ef9-kube-api-access-rm844\") pod \"0677eeba-3e54-4b81-a294-38435cc77ef9\" (UID: \"0677eeba-3e54-4b81-a294-38435cc77ef9\") " Oct 30 19:17:14.402802 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:14.402741 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bbecb8e-40ce-4b0d-948f-c7b53598da3f-kube-api-access-gt8lw\" (UniqueName: \"kubernetes.io/projected/5bbecb8e-40ce-4b0d-948f-c7b53598da3f-kube-api-access-gt8lw\") pod \"5bbecb8e-40ce-4b0d-948f-c7b53598da3f\" (UID: \"5bbecb8e-40ce-4b0d-948f-c7b53598da3f\") " Oct 30 19:17:14.402860 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:14.402810 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bbecb8e-40ce-4b0d-948f-c7b53598da3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bbecb8e-40ce-4b0d-948f-c7b53598da3f-plugins-config-volume\") pod \"5bbecb8e-40ce-4b0d-948f-c7b53598da3f\" (UID: \"5bbecb8e-40ce-4b0d-948f-c7b53598da3f\") " Oct 30 19:17:14.402940 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:14.402903 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0677eeba-3e54-4b81-a294-38435cc77ef9-plugins-config-volume" pod "0677eeba-3e54-4b81-a294-38435cc77ef9" (UID: "0677eeba-3e54-4b81-a294-38435cc77ef9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:14.403232 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:14.403211 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bbecb8e-40ce-4b0d-948f-c7b53598da3f-plugins-config-volume" pod "5bbecb8e-40ce-4b0d-948f-c7b53598da3f" (UID: "5bbecb8e-40ce-4b0d-948f-c7b53598da3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:14.403291 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:14.403221 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0677eeba-3e54-4b81-a294-38435cc77ef9-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:14.404918 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:14.404899 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bbecb8e-40ce-4b0d-948f-c7b53598da3f-kube-api-access-gt8lw" pod "5bbecb8e-40ce-4b0d-948f-c7b53598da3f" (UID: "5bbecb8e-40ce-4b0d-948f-c7b53598da3f"). InnerVolumeSpecName "kube-api-access-gt8lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:14.404918 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:14.404911 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0677eeba-3e54-4b81-a294-38435cc77ef9-kube-api-access-rm844" pod "0677eeba-3e54-4b81-a294-38435cc77ef9" (UID: "0677eeba-3e54-4b81-a294-38435cc77ef9"). InnerVolumeSpecName "kube-api-access-rm844". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:14.504727 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:14.504688 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm844\" (UniqueName: \"kubernetes.io/projected/0677eeba-3e54-4b81-a294-38435cc77ef9-kube-api-access-rm844\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:14.504727 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:14.504720 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt8lw\" (UniqueName: \"kubernetes.io/projected/5bbecb8e-40ce-4b0d-948f-c7b53598da3f-kube-api-access-gt8lw\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:14.504727 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:14.504733 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bbecb8e-40ce-4b0d-948f-c7b53598da3f-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:15.288087 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:15.288051 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 30 19:17:15.288593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:15.288110 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" podUID="f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 30 19:17:15.298652 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:15.298629 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62" Oct 30 19:17:15.298814 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:15.298632 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62" event={"ID":"5bbecb8e-40ce-4b0d-948f-c7b53598da3f","Type":"ContainerDied","Data":"fbc874a82ed8da0a7748e8ae21a6e00d0e21091095dc3ba2633d349827bc38c8"} Oct 30 19:17:15.298814 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:15.298792 4531 scope.go:122] "RemoveContainer" containerID="5298e821c3f4ff6dc1e9ac0d9e8830833ac86291d9468266d2eaf04340b22ad1" Oct 30 19:17:15.300178 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:15.300018 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs" Oct 30 19:17:15.300178 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:15.300067 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs" event={"ID":"0677eeba-3e54-4b81-a294-38435cc77ef9","Type":"ContainerDied","Data":"8efe50238b53e3f26367d23a0916631f0d8390a588f906b6f96c502d0bf839fd"} Oct 30 19:17:15.310039 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:15.310018 4531 scope.go:122] "RemoveContainer" containerID="dcbd49ddbc3af98ba7ea895af733ce704522d2d7d988e31bece622b0bf6a7e6c" Oct 30 19:17:15.462193 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:15.462143 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62"] Oct 30 19:17:15.468744 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:15.468720 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f44c7-569cbbbbc4-57g62"] Oct 30 19:17:15.593227 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:15.593121 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs"] Oct 30 19:17:15.597768 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:15.597739 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f44c7-784796c8c9-tpxxs"] Oct 30 19:17:15.682576 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:15.682548 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp"] Oct 30 19:17:15.687678 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:15.687652 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7f44c7-t58qp"] Oct 30 19:17:16.099223 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:16.099183 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0677eeba-3e54-4b81-a294-38435cc77ef9" path="/var/lib/kubelet/pods/0677eeba-3e54-4b81-a294-38435cc77ef9/volumes" Oct 30 19:17:16.099782 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:16.099762 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bbecb8e-40ce-4b0d-948f-c7b53598da3f" path="/var/lib/kubelet/pods/5bbecb8e-40ce-4b0d-948f-c7b53598da3f/volumes" Oct 30 19:17:16.100368 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:16.100350 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eecc6d95-8f95-4a8d-9b84-5945d02cd42b" path="/var/lib/kubelet/pods/eecc6d95-8f95-4a8d-9b84-5945d02cd42b/volumes" Oct 30 19:17:16.288676 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:16.288632 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 30 19:17:16.289067 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:16.288713 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" podUID="f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 30 19:17:17.292991 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:17.292964 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:17.293363 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:17.293313 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:17.293924 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:17.293909 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:17:20.162469 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.162424 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d60811e6-4lcsf"] Oct 30 19:17:20.162858 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.162844 4531 cpu_manager.go:414] "containerMap: removing container" podUID="5bbecb8e-40ce-4b0d-948f-c7b53598da3f" containerName="payload-processing" Oct 30 19:17:20.162858 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.162856 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bbecb8e-40ce-4b0d-948f-c7b53598da3f" containerName="payload-processing" Oct 30 19:17:20.162947 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.162870 4531 cpu_manager.go:414] "containerMap: removing container" podUID="8256a12e-d979-4a98-8200-0b3937d26dae" containerName="istio-proxy" Oct 30 19:17:20.162947 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.162876 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8256a12e-d979-4a98-8200-0b3937d26dae" containerName="istio-proxy" Oct 30 19:17:20.162947 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.162895 4531 cpu_manager.go:414] "containerMap: removing container" podUID="0677eeba-3e54-4b81-a294-38435cc77ef9" containerName="payload-pre-processing" Oct 30 19:17:20.162947 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.162900 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0677eeba-3e54-4b81-a294-38435cc77ef9" containerName="payload-pre-processing" Oct 30 19:17:20.162947 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.162910 4531 cpu_manager.go:414] "containerMap: removing container" podUID="3a38212e-aab1-4e15-abd6-23ed981aac8f" containerName="revoke-keys" Oct 30 19:17:20.162947 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.162915 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a38212e-aab1-4e15-abd6-23ed981aac8f" containerName="revoke-keys" Oct 30 19:17:20.163181 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.162979 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0677eeba-3e54-4b81-a294-38435cc77ef9" containerName="payload-pre-processing" Oct 30 19:17:20.163181 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.162989 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a38212e-aab1-4e15-abd6-23ed981aac8f" containerName="revoke-keys" Oct 30 19:17:20.163181 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.162999 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bbecb8e-40ce-4b0d-948f-c7b53598da3f" containerName="payload-processing" Oct 30 19:17:20.163181 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.163007 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8256a12e-d979-4a98-8200-0b3937d26dae" containerName="istio-proxy" Oct 30 19:17:20.167752 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.167731 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60811e6-4lcsf" Oct 30 19:17:20.271742 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.271703 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:17:20.271949 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.271762 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:17:20.271949 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.271780 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:17:20.271949 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.271824 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:17:20.274575 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.274555 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d60811e6-4lcsf"] Oct 30 19:17:20.276405 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.276373 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/258738e9-9836-4b2a-a496-58d2183deb72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d60811e6-4lcsf\" (UID: \"258738e9-9836-4b2a-a496-58d2183deb72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60811e6-4lcsf" Oct 30 19:17:20.276512 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.276418 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/258738e9-9836-4b2a-a496-58d2183deb72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d60811e6-4lcsf\" (UID: \"258738e9-9836-4b2a-a496-58d2183deb72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60811e6-4lcsf" Oct 30 19:17:20.276551 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.276535 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvp4l\" (UniqueName: \"kubernetes.io/projected/258738e9-9836-4b2a-a496-58d2183deb72-kube-api-access-bvp4l\") pod \"maas-api-revoke-sub-d60811e6-4lcsf\" (UID: \"258738e9-9836-4b2a-a496-58d2183deb72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60811e6-4lcsf" Oct 30 19:17:20.377431 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.377388 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvp4l\" (UniqueName: \"kubernetes.io/projected/258738e9-9836-4b2a-a496-58d2183deb72-kube-api-access-bvp4l\") pod \"maas-api-revoke-sub-d60811e6-4lcsf\" (UID: \"258738e9-9836-4b2a-a496-58d2183deb72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60811e6-4lcsf" Oct 30 19:17:20.377656 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.377499 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/258738e9-9836-4b2a-a496-58d2183deb72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d60811e6-4lcsf\" (UID: \"258738e9-9836-4b2a-a496-58d2183deb72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60811e6-4lcsf" Oct 30 19:17:20.377656 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.377526 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/258738e9-9836-4b2a-a496-58d2183deb72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d60811e6-4lcsf\" (UID: \"258738e9-9836-4b2a-a496-58d2183deb72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60811e6-4lcsf" Oct 30 19:17:20.378158 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.378121 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/258738e9-9836-4b2a-a496-58d2183deb72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d60811e6-4lcsf\" (UID: \"258738e9-9836-4b2a-a496-58d2183deb72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60811e6-4lcsf" Oct 30 19:17:20.380052 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.380019 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/258738e9-9836-4b2a-a496-58d2183deb72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d60811e6-4lcsf\" (UID: \"258738e9-9836-4b2a-a496-58d2183deb72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60811e6-4lcsf" Oct 30 19:17:20.439624 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.439547 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvp4l\" (UniqueName: \"kubernetes.io/projected/258738e9-9836-4b2a-a496-58d2183deb72-kube-api-access-bvp4l\") pod \"maas-api-revoke-sub-d60811e6-4lcsf\" (UID: \"258738e9-9836-4b2a-a496-58d2183deb72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60811e6-4lcsf" Oct 30 19:17:20.481262 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.481231 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60811e6-4lcsf" Oct 30 19:17:20.868315 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:20.868291 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d60811e6-4lcsf"] Oct 30 19:17:20.870380 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:17:20.870350 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod258738e9_9836_4b2a_a496_58d2183deb72.slice/crio-0a187ac8fbd7e6cfcb56141f585df1d7f2605963a8339166451de2668858694d WatchSource:0}: Error finding container 0a187ac8fbd7e6cfcb56141f585df1d7f2605963a8339166451de2668858694d: Status 404 returned error can't find the container with id 0a187ac8fbd7e6cfcb56141f585df1d7f2605963a8339166451de2668858694d Oct 30 19:17:21.181798 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:21.181748 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c"] Oct 30 19:17:21.185536 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:21.185512 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c" Oct 30 19:17:21.242637 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:21.242602 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-91cb32-dockercfg-shpfn\"" Oct 30 19:17:21.246378 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:21.246341 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c"] Oct 30 19:17:21.287554 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:21.287518 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c\" (UID: \"18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c" Oct 30 19:17:21.287744 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:21.287575 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btqv4\" (UniqueName: \"kubernetes.io/projected/18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3-kube-api-access-btqv4\") pod \"payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c\" (UID: \"18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c" Oct 30 19:17:21.312291 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:21.312259 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-91cb32\"" Oct 30 19:17:21.339003 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:21.338966 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60811e6-4lcsf" event={"ID":"258738e9-9836-4b2a-a496-58d2183deb72","Type":"ContainerStarted","Data":"906de970a81b1d66943fd4141e91f76c2eb4cdb6d9e841ba24c3344e5a113d9d"} Oct 30 19:17:21.339225 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:21.339023 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60811e6-4lcsf" event={"ID":"258738e9-9836-4b2a-a496-58d2183deb72","Type":"ContainerStarted","Data":"0a187ac8fbd7e6cfcb56141f585df1d7f2605963a8339166451de2668858694d"} Oct 30 19:17:21.388715 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:21.388631 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c\" (UID: \"18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c" Oct 30 19:17:21.388715 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:21.388692 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btqv4\" (UniqueName: \"kubernetes.io/projected/18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3-kube-api-access-btqv4\") pod \"payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c\" (UID: \"18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c" Oct 30 19:17:21.389383 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:21.389351 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c\" (UID: \"18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c" Oct 30 19:17:21.450058 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:21.450023 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btqv4\" (UniqueName: \"kubernetes.io/projected/18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3-kube-api-access-btqv4\") pod \"payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c\" (UID: \"18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c" Oct 30 19:17:21.455123 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:21.455077 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60811e6-4lcsf" podStartSLOduration=1.4550596040000001 podStartE2EDuration="1.455059604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:21.454370815 +0000 UTC m=+2601.481475776" watchObservedRunningTime="2026-10-30 19:17:21.455059604 +0000 UTC m=+2601.482164562" Oct 30 19:17:21.498847 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:21.498807 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c" Oct 30 19:17:21.885978 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:21.885949 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c"] Oct 30 19:17:21.887328 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:17:21.887302 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18bc7546_55ad_4f84_9eb6_4e2b3cf1c4f3.slice/crio-59a47501acfc44dd2192f041bd129b8f1073828f29fc3e23e6fedbe929407b8f WatchSource:0}: Error finding container 59a47501acfc44dd2192f041bd129b8f1073828f29fc3e23e6fedbe929407b8f: Status 404 returned error can't find the container with id 59a47501acfc44dd2192f041bd129b8f1073828f29fc3e23e6fedbe929407b8f Oct 30 19:17:21.981101 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:21.981051 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9"] Oct 30 19:17:21.984804 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:21.984780 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9" Oct 30 19:17:22.044826 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:22.044744 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9"] Oct 30 19:17:22.095486 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:22.095449 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8n42\" (UniqueName: \"kubernetes.io/projected/667fc3fe-bd84-4db3-9ab4-cca263a6d1cb-kube-api-access-z8n42\") pod \"payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9\" (UID: \"667fc3fe-bd84-4db3-9ab4-cca263a6d1cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9" Oct 30 19:17:22.095690 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:22.095576 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/667fc3fe-bd84-4db3-9ab4-cca263a6d1cb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9\" (UID: \"667fc3fe-bd84-4db3-9ab4-cca263a6d1cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9" Oct 30 19:17:22.196949 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:22.196903 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8n42\" (UniqueName: \"kubernetes.io/projected/667fc3fe-bd84-4db3-9ab4-cca263a6d1cb-kube-api-access-z8n42\") pod \"payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9\" (UID: \"667fc3fe-bd84-4db3-9ab4-cca263a6d1cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9" Oct 30 19:17:22.197458 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:22.196982 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/667fc3fe-bd84-4db3-9ab4-cca263a6d1cb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9\" (UID: \"667fc3fe-bd84-4db3-9ab4-cca263a6d1cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9" Oct 30 19:17:22.197743 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:22.197722 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/667fc3fe-bd84-4db3-9ab4-cca263a6d1cb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9\" (UID: \"667fc3fe-bd84-4db3-9ab4-cca263a6d1cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9" Oct 30 19:17:22.258049 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:22.258012 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8n42\" (UniqueName: \"kubernetes.io/projected/667fc3fe-bd84-4db3-9ab4-cca263a6d1cb-kube-api-access-z8n42\") pod \"payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9\" (UID: \"667fc3fe-bd84-4db3-9ab4-cca263a6d1cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9" Oct 30 19:17:22.298944 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:22.298843 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9" Oct 30 19:17:22.348731 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:22.348700 4531 generic.go:356] "Generic (PLEG): container finished" podID="258738e9-9836-4b2a-a496-58d2183deb72" containerID="906de970a81b1d66943fd4141e91f76c2eb4cdb6d9e841ba24c3344e5a113d9d" exitCode=0 Oct 30 19:17:22.348911 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:22.348794 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60811e6-4lcsf" event={"ID":"258738e9-9836-4b2a-a496-58d2183deb72","Type":"ContainerDied","Data":"906de970a81b1d66943fd4141e91f76c2eb4cdb6d9e841ba24c3344e5a113d9d"} Oct 30 19:17:22.353360 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:22.351341 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c" event={"ID":"18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3","Type":"ContainerStarted","Data":"e27f2f718e6d0f1a64f1706e585d8aea6e12468c2954ba9eda2c2526302ec8bf"} Oct 30 19:17:22.353360 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:22.351398 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c" event={"ID":"18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3","Type":"ContainerStarted","Data":"59a47501acfc44dd2192f041bd129b8f1073828f29fc3e23e6fedbe929407b8f"} Oct 30 19:17:22.353360 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:22.352055 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c" Oct 30 19:17:22.581889 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:22.581769 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c" podStartSLOduration=1.5817523279999999 podStartE2EDuration="1.581752328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:22.58026829 +0000 UTC m=+2602.607373241" watchObservedRunningTime="2026-10-30 19:17:22.581752328 +0000 UTC m=+2602.608857326" Oct 30 19:17:22.677860 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:22.677833 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9"] Oct 30 19:17:22.679163 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:17:22.679114 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667fc3fe_bd84_4db3_9ab4_cca263a6d1cb.slice/crio-3882dad77f85ce0e510cdd160b3807b95b1b7d566a6c8f14908e73e1b5f14403 WatchSource:0}: Error finding container 3882dad77f85ce0e510cdd160b3807b95b1b7d566a6c8f14908e73e1b5f14403: Status 404 returned error can't find the container with id 3882dad77f85ce0e510cdd160b3807b95b1b7d566a6c8f14908e73e1b5f14403 Oct 30 19:17:23.360776 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:23.360734 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9" event={"ID":"667fc3fe-bd84-4db3-9ab4-cca263a6d1cb","Type":"ContainerStarted","Data":"91fbd3109eeaf0d90bd4ca362379445aee7a5989375cffa3074050dcb8e6fc5a"} Oct 30 19:17:23.361327 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:23.360804 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9" event={"ID":"667fc3fe-bd84-4db3-9ab4-cca263a6d1cb","Type":"ContainerStarted","Data":"3882dad77f85ce0e510cdd160b3807b95b1b7d566a6c8f14908e73e1b5f14403"} Oct 30 19:17:23.361327 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:23.360952 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9" Oct 30 19:17:23.483673 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:23.483592 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9" podStartSLOduration=2.483567203 podStartE2EDuration="2.483567203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:23.479332383 +0000 UTC m=+2603.506437368" watchObservedRunningTime="2026-10-30 19:17:23.483567203 +0000 UTC m=+2603.510672165" Oct 30 19:17:23.525406 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:23.525360 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60811e6-4lcsf" Oct 30 19:17:23.613396 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:23.613290 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/258738e9-9836-4b2a-a496-58d2183deb72-kube-api-access-bvp4l\" (UniqueName: \"kubernetes.io/projected/258738e9-9836-4b2a-a496-58d2183deb72-kube-api-access-bvp4l\") pod \"258738e9-9836-4b2a-a496-58d2183deb72\" (UID: \"258738e9-9836-4b2a-a496-58d2183deb72\") " Oct 30 19:17:23.613396 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:23.613333 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/258738e9-9836-4b2a-a496-58d2183deb72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/258738e9-9836-4b2a-a496-58d2183deb72-maas-api-xks-ca\") pod \"258738e9-9836-4b2a-a496-58d2183deb72\" (UID: \"258738e9-9836-4b2a-a496-58d2183deb72\") " Oct 30 19:17:23.613396 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:23.613379 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/258738e9-9836-4b2a-a496-58d2183deb72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/258738e9-9836-4b2a-a496-58d2183deb72-maas-api-openshift-ca\") pod \"258738e9-9836-4b2a-a496-58d2183deb72\" (UID: \"258738e9-9836-4b2a-a496-58d2183deb72\") " Oct 30 19:17:23.613867 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:23.613839 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/258738e9-9836-4b2a-a496-58d2183deb72-maas-api-openshift-ca" pod "258738e9-9836-4b2a-a496-58d2183deb72" (UID: "258738e9-9836-4b2a-a496-58d2183deb72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:23.615898 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:23.615862 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/258738e9-9836-4b2a-a496-58d2183deb72-maas-api-xks-ca" pod "258738e9-9836-4b2a-a496-58d2183deb72" (UID: "258738e9-9836-4b2a-a496-58d2183deb72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:23.616067 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:23.615937 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/258738e9-9836-4b2a-a496-58d2183deb72-kube-api-access-bvp4l" pod "258738e9-9836-4b2a-a496-58d2183deb72" (UID: "258738e9-9836-4b2a-a496-58d2183deb72"). InnerVolumeSpecName "kube-api-access-bvp4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:23.714517 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:23.714473 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvp4l\" (UniqueName: \"kubernetes.io/projected/258738e9-9836-4b2a-a496-58d2183deb72-kube-api-access-bvp4l\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:23.714517 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:23.714508 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/258738e9-9836-4b2a-a496-58d2183deb72-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:23.714517 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:23.714518 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/258738e9-9836-4b2a-a496-58d2183deb72-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:24.369716 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:24.369631 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60811e6-4lcsf" Oct 30 19:17:24.369716 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:24.369661 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60811e6-4lcsf" event={"ID":"258738e9-9836-4b2a-a496-58d2183deb72","Type":"ContainerDied","Data":"0a187ac8fbd7e6cfcb56141f585df1d7f2605963a8339166451de2668858694d"} Oct 30 19:17:24.369716 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:24.369714 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a187ac8fbd7e6cfcb56141f585df1d7f2605963a8339166451de2668858694d" Oct 30 19:17:26.408756 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:17:26.408708 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:17:34.371519 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:34.371490 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9" Oct 30 19:17:34.371957 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:34.371918 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c" Oct 30 19:17:39.221646 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.221608 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh"] Oct 30 19:17:39.222030 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.222016 4531 cpu_manager.go:414] "containerMap: removing container" podUID="258738e9-9836-4b2a-a496-58d2183deb72" containerName="revoke-keys" Oct 30 19:17:39.222092 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.222031 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="258738e9-9836-4b2a-a496-58d2183deb72" containerName="revoke-keys" Oct 30 19:17:39.222156 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.222111 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="258738e9-9836-4b2a-a496-58d2183deb72" containerName="revoke-keys" Oct 30 19:17:39.225449 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.225427 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.270919 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.270886 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74115f83-42g89"] Oct 30 19:17:39.274429 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.274412 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74115f83-42g89" Oct 30 19:17:39.288221 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.288193 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0832817-121a-44cb-b8a1-81c4446f6aea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74115f83-42g89\" (UID: \"f0832817-121a-44cb-b8a1-81c4446f6aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74115f83-42g89" Oct 30 19:17:39.288432 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.288409 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0832817-121a-44cb-b8a1-81c4446f6aea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74115f83-42g89\" (UID: \"f0832817-121a-44cb-b8a1-81c4446f6aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74115f83-42g89" Oct 30 19:17:39.288502 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.288473 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt66n\" (UniqueName: \"kubernetes.io/projected/f0832817-121a-44cb-b8a1-81c4446f6aea-kube-api-access-tt66n\") pod \"maas-api-revoke-sub-74115f83-42g89\" (UID: \"f0832817-121a-44cb-b8a1-81c4446f6aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74115f83-42g89" Oct 30 19:17:39.328972 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.328936 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:17:39.329180 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.329005 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:17:39.329180 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.329121 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:17:39.329274 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.329194 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:17:39.329904 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.329883 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74115f83-42g89"] Oct 30 19:17:39.333424 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.333396 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh"] Oct 30 19:17:39.389468 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.389430 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0832817-121a-44cb-b8a1-81c4446f6aea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74115f83-42g89\" (UID: \"f0832817-121a-44cb-b8a1-81c4446f6aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74115f83-42g89" Oct 30 19:17:39.389688 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.389473 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5987b871-3251-4621-b79e-198239ddce66-istiod-ca-cert\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.389688 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.389600 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-workload-certs\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.389793 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.389697 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-workload-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.389793 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.389730 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-credential-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.389873 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.389822 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt66n\" (UniqueName: \"kubernetes.io/projected/f0832817-121a-44cb-b8a1-81c4446f6aea-kube-api-access-tt66n\") pod \"maas-api-revoke-sub-74115f83-42g89\" (UID: \"f0832817-121a-44cb-b8a1-81c4446f6aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74115f83-42g89" Oct 30 19:17:39.389873 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.389851 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-istio-envoy\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.389873 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.389867 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-istio-data\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.390025 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.389891 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c4f7\" (UniqueName: \"kubernetes.io/projected/5987b871-3251-4621-b79e-198239ddce66-kube-api-access-2c4f7\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.390025 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.389915 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5987b871-3251-4621-b79e-198239ddce66-istio-podinfo\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.390025 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.390008 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0832817-121a-44cb-b8a1-81c4446f6aea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74115f83-42g89\" (UID: \"f0832817-121a-44cb-b8a1-81c4446f6aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74115f83-42g89" Oct 30 19:17:39.390193 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.390078 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5987b871-3251-4621-b79e-198239ddce66-istio-token\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.390193 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.390124 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0832817-121a-44cb-b8a1-81c4446f6aea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74115f83-42g89\" (UID: \"f0832817-121a-44cb-b8a1-81c4446f6aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74115f83-42g89" Oct 30 19:17:39.392429 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.392406 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0832817-121a-44cb-b8a1-81c4446f6aea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74115f83-42g89\" (UID: \"f0832817-121a-44cb-b8a1-81c4446f6aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74115f83-42g89" Oct 30 19:17:39.451470 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.451432 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt66n\" (UniqueName: \"kubernetes.io/projected/f0832817-121a-44cb-b8a1-81c4446f6aea-kube-api-access-tt66n\") pod \"maas-api-revoke-sub-74115f83-42g89\" (UID: \"f0832817-121a-44cb-b8a1-81c4446f6aea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74115f83-42g89" Oct 30 19:17:39.490830 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.490743 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5987b871-3251-4621-b79e-198239ddce66-istio-token\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.490830 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.490811 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5987b871-3251-4621-b79e-198239ddce66-istiod-ca-cert\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.491071 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.490839 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-workload-certs\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.491071 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.490868 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-workload-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.491071 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.490891 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-credential-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.491071 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.490962 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-istio-envoy\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.491071 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.491020 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-istio-data\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.491386 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.491071 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c4f7\" (UniqueName: \"kubernetes.io/projected/5987b871-3251-4621-b79e-198239ddce66-kube-api-access-2c4f7\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.491386 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.491106 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5987b871-3251-4621-b79e-198239ddce66-istio-podinfo\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.491557 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.491458 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-workload-certs\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.491643 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.491616 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5987b871-3251-4621-b79e-198239ddce66-istiod-ca-cert\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.491738 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.491703 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-istio-data\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.491810 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.491737 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-credential-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.491978 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.491960 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-workload-socket\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.493736 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.493715 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5987b871-3251-4621-b79e-198239ddce66-istio-podinfo\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.493810 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.493763 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-istio-envoy\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.553267 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.553221 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5987b871-3251-4621-b79e-198239ddce66-istio-token\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.553945 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.553914 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c4f7\" (UniqueName: \"kubernetes.io/projected/5987b871-3251-4621-b79e-198239ddce66-kube-api-access-2c4f7\") pod \"e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.556977 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.556952 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl"] Oct 30 19:17:39.563306 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.563283 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl" Oct 30 19:17:39.587215 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.587177 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74115f83-42g89" Oct 30 19:17:39.591702 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.591676 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vttgf\" (UniqueName: \"kubernetes.io/projected/5364e815-b782-44fc-a104-b05aabc0c3d6-kube-api-access-vttgf\") pod \"maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl\" (UID: \"5364e815-b782-44fc-a104-b05aabc0c3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl" Oct 30 19:17:39.591816 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.591742 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5364e815-b782-44fc-a104-b05aabc0c3d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl\" (UID: \"5364e815-b782-44fc-a104-b05aabc0c3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl" Oct 30 19:17:39.591874 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.591813 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5364e815-b782-44fc-a104-b05aabc0c3d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl\" (UID: \"5364e815-b782-44fc-a104-b05aabc0c3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl" Oct 30 19:17:39.620165 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.620093 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl"] Oct 30 19:17:39.692826 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.692781 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vttgf\" (UniqueName: \"kubernetes.io/projected/5364e815-b782-44fc-a104-b05aabc0c3d6-kube-api-access-vttgf\") pod \"maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl\" (UID: \"5364e815-b782-44fc-a104-b05aabc0c3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl" Oct 30 19:17:39.693057 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.692868 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5364e815-b782-44fc-a104-b05aabc0c3d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl\" (UID: \"5364e815-b782-44fc-a104-b05aabc0c3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl" Oct 30 19:17:39.693057 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.692926 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5364e815-b782-44fc-a104-b05aabc0c3d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl\" (UID: \"5364e815-b782-44fc-a104-b05aabc0c3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl" Oct 30 19:17:39.693872 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.693838 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5364e815-b782-44fc-a104-b05aabc0c3d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl\" (UID: \"5364e815-b782-44fc-a104-b05aabc0c3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl" Oct 30 19:17:39.696202 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.696178 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5364e815-b782-44fc-a104-b05aabc0c3d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl\" (UID: \"5364e815-b782-44fc-a104-b05aabc0c3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl" Oct 30 19:17:39.754770 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.754737 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vttgf\" (UniqueName: \"kubernetes.io/projected/5364e815-b782-44fc-a104-b05aabc0c3d6-kube-api-access-vttgf\") pod \"maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl\" (UID: \"5364e815-b782-44fc-a104-b05aabc0c3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl" Oct 30 19:17:39.839423 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.839382 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:39.878329 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.878289 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl" Oct 30 19:17:39.973691 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:39.973654 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74115f83-42g89"] Oct 30 19:17:39.978945 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:17:39.978908 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0832817_121a_44cb_b8a1_81c4446f6aea.slice/crio-758cc261e27b196f501963b1e5e76f24f0a997622b3db2fedb93f5a972180373 WatchSource:0}: Error finding container 758cc261e27b196f501963b1e5e76f24f0a997622b3db2fedb93f5a972180373: Status 404 returned error can't find the container with id 758cc261e27b196f501963b1e5e76f24f0a997622b3db2fedb93f5a972180373 Oct 30 19:17:40.223115 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:40.223086 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh"] Oct 30 19:17:40.224935 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:17:40.224907 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5987b871_3251_4621_b79e_198239ddce66.slice/crio-9d917bb0b27d493f319ba885e28f046087b62d15538192cdedb2e55108f9b42b WatchSource:0}: Error finding container 9d917bb0b27d493f319ba885e28f046087b62d15538192cdedb2e55108f9b42b: Status 404 returned error can't find the container with id 9d917bb0b27d493f319ba885e28f046087b62d15538192cdedb2e55108f9b42b Oct 30 19:17:40.227681 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:40.227634 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:17:40.227778 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:40.227725 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:17:40.227778 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:40.227767 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:17:40.278367 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:40.278339 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl"] Oct 30 19:17:40.279747 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:17:40.279715 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5364e815_b782_44fc_a104_b05aabc0c3d6.slice/crio-58ec82a8c4c44a559dbd5980411471340a60fbfb04c68196d7feac634a6fcdc1 WatchSource:0}: Error finding container 58ec82a8c4c44a559dbd5980411471340a60fbfb04c68196d7feac634a6fcdc1: Status 404 returned error can't find the container with id 58ec82a8c4c44a559dbd5980411471340a60fbfb04c68196d7feac634a6fcdc1 Oct 30 19:17:40.476848 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:40.476811 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl" event={"ID":"5364e815-b782-44fc-a104-b05aabc0c3d6","Type":"ContainerStarted","Data":"58ec82a8c4c44a559dbd5980411471340a60fbfb04c68196d7feac634a6fcdc1"} Oct 30 19:17:40.478614 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:40.478589 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" event={"ID":"5987b871-3251-4621-b79e-198239ddce66","Type":"ContainerStarted","Data":"90934e5b573a94e33eda67ec31f9e64e4c678ba53e7b5be3f072376d9cb11ebd"} Oct 30 19:17:40.478749 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:40.478634 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" event={"ID":"5987b871-3251-4621-b79e-198239ddce66","Type":"ContainerStarted","Data":"9d917bb0b27d493f319ba885e28f046087b62d15538192cdedb2e55108f9b42b"} Oct 30 19:17:40.480233 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:40.480212 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74115f83-42g89" event={"ID":"f0832817-121a-44cb-b8a1-81c4446f6aea","Type":"ContainerStarted","Data":"5b9c8f25496b002ed36b413fe2f0a5791b1cae13632229907db4274861846c66"} Oct 30 19:17:40.480325 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:40.480245 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74115f83-42g89" event={"ID":"f0832817-121a-44cb-b8a1-81c4446f6aea","Type":"ContainerStarted","Data":"758cc261e27b196f501963b1e5e76f24f0a997622b3db2fedb93f5a972180373"} Oct 30 19:17:40.603795 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:40.603734 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" podStartSLOduration=1.603718503 podStartE2EDuration="1.603718503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:40.601917513 +0000 UTC m=+2620.629022473" watchObservedRunningTime="2026-10-30 19:17:40.603718503 +0000 UTC m=+2620.630823461" Oct 30 19:17:40.732286 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:40.729641 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74115f83-42g89" podStartSLOduration=1.729615011 podStartE2EDuration="1.729615011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:40.722616618 +0000 UTC m=+2620.749721591" watchObservedRunningTime="2026-10-30 19:17:40.729615011 +0000 UTC m=+2620.756719989" Oct 30 19:17:40.840084 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:40.839993 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:41.488689 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:41.488594 4531 generic.go:356] "Generic (PLEG): container finished" podID="f0832817-121a-44cb-b8a1-81c4446f6aea" containerID="5b9c8f25496b002ed36b413fe2f0a5791b1cae13632229907db4274861846c66" exitCode=0 Oct 30 19:17:41.489099 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:41.488689 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74115f83-42g89" event={"ID":"f0832817-121a-44cb-b8a1-81c4446f6aea","Type":"ContainerDied","Data":"5b9c8f25496b002ed36b413fe2f0a5791b1cae13632229907db4274861846c66"} Oct 30 19:17:41.490495 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:41.490461 4531 generic.go:356] "Generic (PLEG): container finished" podID="5364e815-b782-44fc-a104-b05aabc0c3d6" containerID="954cccde6c438f7fad55efcc734bb71cdc90ec3e3feba1ee8875ac0a60c63009" exitCode=0 Oct 30 19:17:41.490632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:41.490533 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl" event={"ID":"5364e815-b782-44fc-a104-b05aabc0c3d6","Type":"ContainerDied","Data":"954cccde6c438f7fad55efcc734bb71cdc90ec3e3feba1ee8875ac0a60c63009"} Oct 30 19:17:41.842419 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:41.842377 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 19:17:41.842625 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:41.842451 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" podUID="5987b871-3251-4621-b79e-198239ddce66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 19:17:42.752954 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:42.752885 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74115f83-42g89" Oct 30 19:17:42.756719 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:42.756689 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl" Oct 30 19:17:42.840191 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:42.839952 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 19:17:42.840191 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:42.840020 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" podUID="5987b871-3251-4621-b79e-198239ddce66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 19:17:42.855272 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:42.855233 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0832817-121a-44cb-b8a1-81c4446f6aea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0832817-121a-44cb-b8a1-81c4446f6aea-maas-api-xks-ca\") pod \"f0832817-121a-44cb-b8a1-81c4446f6aea\" (UID: \"f0832817-121a-44cb-b8a1-81c4446f6aea\") " Oct 30 19:17:42.855461 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:42.855303 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0832817-121a-44cb-b8a1-81c4446f6aea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0832817-121a-44cb-b8a1-81c4446f6aea-maas-api-openshift-ca\") pod \"f0832817-121a-44cb-b8a1-81c4446f6aea\" (UID: \"f0832817-121a-44cb-b8a1-81c4446f6aea\") " Oct 30 19:17:42.855461 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:42.855351 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5364e815-b782-44fc-a104-b05aabc0c3d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5364e815-b782-44fc-a104-b05aabc0c3d6-maas-api-xks-ca\") pod \"5364e815-b782-44fc-a104-b05aabc0c3d6\" (UID: \"5364e815-b782-44fc-a104-b05aabc0c3d6\") " Oct 30 19:17:42.855461 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:42.855392 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5364e815-b782-44fc-a104-b05aabc0c3d6-kube-api-access-vttgf\" (UniqueName: \"kubernetes.io/projected/5364e815-b782-44fc-a104-b05aabc0c3d6-kube-api-access-vttgf\") pod \"5364e815-b782-44fc-a104-b05aabc0c3d6\" (UID: \"5364e815-b782-44fc-a104-b05aabc0c3d6\") " Oct 30 19:17:42.855563 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:42.855500 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5364e815-b782-44fc-a104-b05aabc0c3d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5364e815-b782-44fc-a104-b05aabc0c3d6-maas-api-openshift-ca\") pod \"5364e815-b782-44fc-a104-b05aabc0c3d6\" (UID: \"5364e815-b782-44fc-a104-b05aabc0c3d6\") " Oct 30 19:17:42.855563 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:42.855543 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0832817-121a-44cb-b8a1-81c4446f6aea-kube-api-access-tt66n\" (UniqueName: \"kubernetes.io/projected/f0832817-121a-44cb-b8a1-81c4446f6aea-kube-api-access-tt66n\") pod \"f0832817-121a-44cb-b8a1-81c4446f6aea\" (UID: \"f0832817-121a-44cb-b8a1-81c4446f6aea\") " Oct 30 19:17:42.855699 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:42.855667 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0832817-121a-44cb-b8a1-81c4446f6aea-maas-api-openshift-ca" pod "f0832817-121a-44cb-b8a1-81c4446f6aea" (UID: "f0832817-121a-44cb-b8a1-81c4446f6aea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:42.856009 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:42.855986 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0832817-121a-44cb-b8a1-81c4446f6aea-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:42.856350 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:42.856229 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5364e815-b782-44fc-a104-b05aabc0c3d6-maas-api-openshift-ca" pod "5364e815-b782-44fc-a104-b05aabc0c3d6" (UID: "5364e815-b782-44fc-a104-b05aabc0c3d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:42.857787 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:42.857763 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5364e815-b782-44fc-a104-b05aabc0c3d6-maas-api-xks-ca" pod "5364e815-b782-44fc-a104-b05aabc0c3d6" (UID: "5364e815-b782-44fc-a104-b05aabc0c3d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:42.857953 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:42.857931 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5364e815-b782-44fc-a104-b05aabc0c3d6-kube-api-access-vttgf" pod "5364e815-b782-44fc-a104-b05aabc0c3d6" (UID: "5364e815-b782-44fc-a104-b05aabc0c3d6"). InnerVolumeSpecName "kube-api-access-vttgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:42.858091 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:42.857990 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0832817-121a-44cb-b8a1-81c4446f6aea-maas-api-xks-ca" pod "f0832817-121a-44cb-b8a1-81c4446f6aea" (UID: "f0832817-121a-44cb-b8a1-81c4446f6aea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:42.858284 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:42.858259 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0832817-121a-44cb-b8a1-81c4446f6aea-kube-api-access-tt66n" pod "f0832817-121a-44cb-b8a1-81c4446f6aea" (UID: "f0832817-121a-44cb-b8a1-81c4446f6aea"). InnerVolumeSpecName "kube-api-access-tt66n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:42.957114 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:42.957070 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5364e815-b782-44fc-a104-b05aabc0c3d6-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:42.957114 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:42.957114 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vttgf\" (UniqueName: \"kubernetes.io/projected/5364e815-b782-44fc-a104-b05aabc0c3d6-kube-api-access-vttgf\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:42.957363 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:42.957152 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5364e815-b782-44fc-a104-b05aabc0c3d6-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:42.957363 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:42.957161 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt66n\" (UniqueName: \"kubernetes.io/projected/f0832817-121a-44cb-b8a1-81c4446f6aea-kube-api-access-tt66n\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:42.957363 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:42.957170 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0832817-121a-44cb-b8a1-81c4446f6aea-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:43.505421 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:43.505387 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74115f83-42g89" Oct 30 19:17:43.505421 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:43.505413 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74115f83-42g89" event={"ID":"f0832817-121a-44cb-b8a1-81c4446f6aea","Type":"ContainerDied","Data":"758cc261e27b196f501963b1e5e76f24f0a997622b3db2fedb93f5a972180373"} Oct 30 19:17:43.505712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:43.505469 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="758cc261e27b196f501963b1e5e76f24f0a997622b3db2fedb93f5a972180373" Oct 30 19:17:43.506953 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:43.506930 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl" Oct 30 19:17:43.507105 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:43.506952 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl" event={"ID":"5364e815-b782-44fc-a104-b05aabc0c3d6","Type":"ContainerDied","Data":"58ec82a8c4c44a559dbd5980411471340a60fbfb04c68196d7feac634a6fcdc1"} Oct 30 19:17:43.507105 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:43.507003 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58ec82a8c4c44a559dbd5980411471340a60fbfb04c68196d7feac634a6fcdc1" Oct 30 19:17:43.840716 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:43.840612 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 19:17:43.840716 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:43.840675 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" podUID="5987b871-3251-4621-b79e-198239ddce66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 19:17:44.840696 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:44.840661 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 19:17:44.840911 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:44.840715 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" podUID="5987b871-3251-4621-b79e-198239ddce66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 19:17:45.840627 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:45.840589 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 19:17:45.840810 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:45.840655 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" podUID="5987b871-3251-4621-b79e-198239ddce66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 19:17:46.840504 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:46.840462 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 19:17:46.840934 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:46.840517 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" podUID="5987b871-3251-4621-b79e-198239ddce66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 19:17:47.844198 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:47.844164 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:47.844678 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:47.844413 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:47.845272 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:47.845255 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:17:52.760695 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:52.760652 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h"] Oct 30 19:17:52.761222 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:52.760901 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h" podUID="4cefcca3-b31c-4b38-9995-650aa594f2cd" containerName="payload-processing" containerID="cri-o://7c7e513e31687daa1f37a14a961c745e811cf32a3759d04f9521ba5f76b19b5b" gracePeriod=30 Oct 30 19:17:53.023672 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.023606 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h" Oct 30 19:17:53.092872 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.092835 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cefcca3-b31c-4b38-9995-650aa594f2cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cefcca3-b31c-4b38-9995-650aa594f2cd-plugins-config-volume\") pod \"4cefcca3-b31c-4b38-9995-650aa594f2cd\" (UID: \"4cefcca3-b31c-4b38-9995-650aa594f2cd\") " Oct 30 19:17:53.093077 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.092890 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cefcca3-b31c-4b38-9995-650aa594f2cd-kube-api-access-tr44w\" (UniqueName: \"kubernetes.io/projected/4cefcca3-b31c-4b38-9995-650aa594f2cd-kube-api-access-tr44w\") pod \"4cefcca3-b31c-4b38-9995-650aa594f2cd\" (UID: \"4cefcca3-b31c-4b38-9995-650aa594f2cd\") " Oct 30 19:17:53.093271 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.093242 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cefcca3-b31c-4b38-9995-650aa594f2cd-plugins-config-volume" pod "4cefcca3-b31c-4b38-9995-650aa594f2cd" (UID: "4cefcca3-b31c-4b38-9995-650aa594f2cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:53.095112 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.095088 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cefcca3-b31c-4b38-9995-650aa594f2cd-kube-api-access-tr44w" pod "4cefcca3-b31c-4b38-9995-650aa594f2cd" (UID: "4cefcca3-b31c-4b38-9995-650aa594f2cd"). InnerVolumeSpecName "kube-api-access-tr44w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:53.162438 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.162385 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cca9728-jtdjp"] Oct 30 19:17:53.163001 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.162985 4531 cpu_manager.go:414] "containerMap: removing container" podUID="f0832817-121a-44cb-b8a1-81c4446f6aea" containerName="revoke-keys" Oct 30 19:17:53.163063 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.163004 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0832817-121a-44cb-b8a1-81c4446f6aea" containerName="revoke-keys" Oct 30 19:17:53.163063 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.163016 4531 cpu_manager.go:414] "containerMap: removing container" podUID="5364e815-b782-44fc-a104-b05aabc0c3d6" containerName="revoke-keys" Oct 30 19:17:53.163063 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.163021 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5364e815-b782-44fc-a104-b05aabc0c3d6" containerName="revoke-keys" Oct 30 19:17:53.163063 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.163049 4531 cpu_manager.go:414] "containerMap: removing container" podUID="4cefcca3-b31c-4b38-9995-650aa594f2cd" containerName="payload-processing" Oct 30 19:17:53.163063 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.163055 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cefcca3-b31c-4b38-9995-650aa594f2cd" containerName="payload-processing" Oct 30 19:17:53.163250 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.163166 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0832817-121a-44cb-b8a1-81c4446f6aea" containerName="revoke-keys" Oct 30 19:17:53.163250 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.163175 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5364e815-b782-44fc-a104-b05aabc0c3d6" containerName="revoke-keys" Oct 30 19:17:53.163250 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.163186 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cefcca3-b31c-4b38-9995-650aa594f2cd" containerName="payload-processing" Oct 30 19:17:53.166358 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.166341 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca9728-jtdjp" Oct 30 19:17:53.194058 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.194027 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cefcca3-b31c-4b38-9995-650aa594f2cd-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:53.194058 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.194057 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr44w\" (UniqueName: \"kubernetes.io/projected/4cefcca3-b31c-4b38-9995-650aa594f2cd-kube-api-access-tr44w\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:53.224461 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.223861 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:17:53.224461 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.223928 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:17:53.224461 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.223975 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:17:53.224461 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.224248 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:17:53.224800 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.224612 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cca9728-jtdjp"] Oct 30 19:17:53.303297 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.303188 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpxfx\" (UniqueName: \"kubernetes.io/projected/47d36714-4e59-4a7e-a853-59f2a40c4fdc-kube-api-access-xpxfx\") pod \"maas-api-revoke-sub-0cca9728-jtdjp\" (UID: \"47d36714-4e59-4a7e-a853-59f2a40c4fdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca9728-jtdjp" Oct 30 19:17:53.303491 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.303361 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47d36714-4e59-4a7e-a853-59f2a40c4fdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cca9728-jtdjp\" (UID: \"47d36714-4e59-4a7e-a853-59f2a40c4fdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca9728-jtdjp" Oct 30 19:17:53.303491 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.303418 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47d36714-4e59-4a7e-a853-59f2a40c4fdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cca9728-jtdjp\" (UID: \"47d36714-4e59-4a7e-a853-59f2a40c4fdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca9728-jtdjp" Oct 30 19:17:53.404745 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.404707 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47d36714-4e59-4a7e-a853-59f2a40c4fdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cca9728-jtdjp\" (UID: \"47d36714-4e59-4a7e-a853-59f2a40c4fdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca9728-jtdjp" Oct 30 19:17:53.404745 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.404758 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47d36714-4e59-4a7e-a853-59f2a40c4fdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cca9728-jtdjp\" (UID: \"47d36714-4e59-4a7e-a853-59f2a40c4fdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca9728-jtdjp" Oct 30 19:17:53.405031 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.404798 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpxfx\" (UniqueName: \"kubernetes.io/projected/47d36714-4e59-4a7e-a853-59f2a40c4fdc-kube-api-access-xpxfx\") pod \"maas-api-revoke-sub-0cca9728-jtdjp\" (UID: \"47d36714-4e59-4a7e-a853-59f2a40c4fdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca9728-jtdjp" Oct 30 19:17:53.405464 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.405440 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47d36714-4e59-4a7e-a853-59f2a40c4fdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cca9728-jtdjp\" (UID: \"47d36714-4e59-4a7e-a853-59f2a40c4fdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca9728-jtdjp" Oct 30 19:17:53.407874 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.407848 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47d36714-4e59-4a7e-a853-59f2a40c4fdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cca9728-jtdjp\" (UID: \"47d36714-4e59-4a7e-a853-59f2a40c4fdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca9728-jtdjp" Oct 30 19:17:53.465892 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.465856 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpxfx\" (UniqueName: \"kubernetes.io/projected/47d36714-4e59-4a7e-a853-59f2a40c4fdc-kube-api-access-xpxfx\") pod \"maas-api-revoke-sub-0cca9728-jtdjp\" (UID: \"47d36714-4e59-4a7e-a853-59f2a40c4fdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca9728-jtdjp" Oct 30 19:17:53.479767 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.479739 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca9728-jtdjp" Oct 30 19:17:53.612515 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.612414 4531 generic.go:356] "Generic (PLEG): container finished" podID="4cefcca3-b31c-4b38-9995-650aa594f2cd" containerID="7c7e513e31687daa1f37a14a961c745e811cf32a3759d04f9521ba5f76b19b5b" exitCode=0 Oct 30 19:17:53.612515 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.612470 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h" Oct 30 19:17:53.612515 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.612500 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h" event={"ID":"4cefcca3-b31c-4b38-9995-650aa594f2cd","Type":"ContainerDied","Data":"7c7e513e31687daa1f37a14a961c745e811cf32a3759d04f9521ba5f76b19b5b"} Oct 30 19:17:53.612827 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.612567 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h" event={"ID":"4cefcca3-b31c-4b38-9995-650aa594f2cd","Type":"ContainerDied","Data":"1bf3dfb4ebd65902dd77f78cf3940e026b9fa69f6fd678f424636840efdfe784"} Oct 30 19:17:53.612827 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.612595 4531 scope.go:122] "RemoveContainer" containerID="7c7e513e31687daa1f37a14a961c745e811cf32a3759d04f9521ba5f76b19b5b" Oct 30 19:17:53.623664 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.623642 4531 scope.go:122] "RemoveContainer" containerID="7c7e513e31687daa1f37a14a961c745e811cf32a3759d04f9521ba5f76b19b5b" Oct 30 19:17:53.624019 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:17:53.624000 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c7e513e31687daa1f37a14a961c745e811cf32a3759d04f9521ba5f76b19b5b\": container with ID starting with 7c7e513e31687daa1f37a14a961c745e811cf32a3759d04f9521ba5f76b19b5b not found: ID does not exist" containerID="7c7e513e31687daa1f37a14a961c745e811cf32a3759d04f9521ba5f76b19b5b" Oct 30 19:17:53.624188 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.624028 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c7e513e31687daa1f37a14a961c745e811cf32a3759d04f9521ba5f76b19b5b"} err="failed to get container status \"7c7e513e31687daa1f37a14a961c745e811cf32a3759d04f9521ba5f76b19b5b\": rpc error: code = NotFound desc = could not find container \"7c7e513e31687daa1f37a14a961c745e811cf32a3759d04f9521ba5f76b19b5b\": container with ID starting with 7c7e513e31687daa1f37a14a961c745e811cf32a3759d04f9521ba5f76b19b5b not found: ID does not exist" Oct 30 19:17:53.799385 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.799334 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h"] Oct 30 19:17:53.811907 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.811872 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-52d383-77644b7cc6-gwr5h"] Oct 30 19:17:53.928276 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:53.928183 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cca9728-jtdjp"] Oct 30 19:17:53.949953 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:17:53.949918 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d36714_4e59_4a7e_a853_59f2a40c4fdc.slice/crio-a8bc34d7959e3274f34e871c93affeabfe133557e2a454ab5f7fa401e279ebd4 WatchSource:0}: Error finding container a8bc34d7959e3274f34e871c93affeabfe133557e2a454ab5f7fa401e279ebd4: Status 404 returned error can't find the container with id a8bc34d7959e3274f34e871c93affeabfe133557e2a454ab5f7fa401e279ebd4 Oct 30 19:17:54.098462 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:54.098423 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cefcca3-b31c-4b38-9995-650aa594f2cd" path="/var/lib/kubelet/pods/4cefcca3-b31c-4b38-9995-650aa594f2cd/volumes" Oct 30 19:17:54.541325 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:54.541258 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl"] Oct 30 19:17:54.546871 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:54.546056 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52d383-nt4jl"] Oct 30 19:17:54.620779 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:54.620737 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca9728-jtdjp" event={"ID":"47d36714-4e59-4a7e-a853-59f2a40c4fdc","Type":"ContainerStarted","Data":"a4e5d8a1af656470020cb36ad248896419c7a3c46a541ddb874751fd5e95ea8c"} Oct 30 19:17:54.620978 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:54.620824 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca9728-jtdjp" event={"ID":"47d36714-4e59-4a7e-a853-59f2a40c4fdc","Type":"ContainerStarted","Data":"a8bc34d7959e3274f34e871c93affeabfe133557e2a454ab5f7fa401e279ebd4"} Oct 30 19:17:54.739921 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:54.739866 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca9728-jtdjp" podStartSLOduration=1.73985123 podStartE2EDuration="1.73985123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:54.737722673 +0000 UTC m=+2634.764827631" watchObservedRunningTime="2026-10-30 19:17:54.73985123 +0000 UTC m=+2634.766956188" Oct 30 19:17:55.629626 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:55.629589 4531 generic.go:356] "Generic (PLEG): container finished" podID="47d36714-4e59-4a7e-a853-59f2a40c4fdc" containerID="a4e5d8a1af656470020cb36ad248896419c7a3c46a541ddb874751fd5e95ea8c" exitCode=0 Oct 30 19:17:55.630049 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:55.629675 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca9728-jtdjp" event={"ID":"47d36714-4e59-4a7e-a853-59f2a40c4fdc","Type":"ContainerDied","Data":"a4e5d8a1af656470020cb36ad248896419c7a3c46a541ddb874751fd5e95ea8c"} Oct 30 19:17:55.909750 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:55.909668 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g"] Oct 30 19:17:55.916733 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:55.916703 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-6d96a6-zz64g"] Oct 30 19:17:55.946756 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:55.946726 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48"] Oct 30 19:17:55.950542 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:55.950522 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.005664 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.005632 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-4a7a52-openshift-default-dockercfg-9p592\"" Oct 30 19:17:56.031119 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.031077 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48"] Oct 30 19:17:56.032648 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.032620 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-workload-socket\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.032808 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.032659 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-envoy\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.032808 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.032764 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-token\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.032935 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.032911 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ecb161-fdf5-4084-9080-0227a442d88a-istiod-ca-cert\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.032994 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.032949 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-podinfo\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.033097 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.033081 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c559f\" (UniqueName: \"kubernetes.io/projected/b5ecb161-fdf5-4084-9080-0227a442d88a-kube-api-access-c559f\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.033188 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.033161 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-workload-certs\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.033240 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.033223 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-credential-socket\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.033312 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.033293 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-data\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.098734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.098698 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5364e815-b782-44fc-a104-b05aabc0c3d6" path="/var/lib/kubelet/pods/5364e815-b782-44fc-a104-b05aabc0c3d6/volumes" Oct 30 19:17:56.099256 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.099234 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e225e735-5256-4078-9c8f-9bfbcde0308e" path="/var/lib/kubelet/pods/e225e735-5256-4078-9c8f-9bfbcde0308e/volumes" Oct 30 19:17:56.134944 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.134900 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-token\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.135174 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.134961 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ecb161-fdf5-4084-9080-0227a442d88a-istiod-ca-cert\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.135174 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.134995 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-podinfo\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.135174 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.135052 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c559f\" (UniqueName: \"kubernetes.io/projected/b5ecb161-fdf5-4084-9080-0227a442d88a-kube-api-access-c559f\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.135174 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.135105 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-workload-certs\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.135174 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.135143 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-credential-socket\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.135462 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.135288 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-data\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.135462 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.135360 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-workload-socket\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.135462 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.135387 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-envoy\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.135621 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.135594 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-credential-socket\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.135685 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.135629 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-workload-certs\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.135748 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.135707 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-data\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.135748 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.135741 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-workload-socket\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.135882 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.135863 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ecb161-fdf5-4084-9080-0227a442d88a-istiod-ca-cert\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.137752 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.137730 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-envoy\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.137855 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.137735 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-podinfo\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.196408 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.196330 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-token\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.196408 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.196381 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c559f\" (UniqueName: \"kubernetes.io/projected/b5ecb161-fdf5-4084-9080-0227a442d88a-kube-api-access-c559f\") pod \"e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.264945 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.264901 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:56.400784 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:17:56.400750 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:17:56.717643 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.717608 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48"] Oct 30 19:17:56.719172 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:17:56.719112 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ecb161_fdf5_4084_9080_0227a442d88a.slice/crio-85e27ec20a9899838da673e3b6e7b09e0b7883b2a245d56b3eaa2df9aa62d4a8 WatchSource:0}: Error finding container 85e27ec20a9899838da673e3b6e7b09e0b7883b2a245d56b3eaa2df9aa62d4a8: Status 404 returned error can't find the container with id 85e27ec20a9899838da673e3b6e7b09e0b7883b2a245d56b3eaa2df9aa62d4a8 Oct 30 19:17:56.722122 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.722082 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:17:56.722280 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.722182 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:17:56.722280 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.722217 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:17:56.771041 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.771011 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca9728-jtdjp" Oct 30 19:17:56.948554 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.948462 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d36714-4e59-4a7e-a853-59f2a40c4fdc-kube-api-access-xpxfx\" (UniqueName: \"kubernetes.io/projected/47d36714-4e59-4a7e-a853-59f2a40c4fdc-kube-api-access-xpxfx\") pod \"47d36714-4e59-4a7e-a853-59f2a40c4fdc\" (UID: \"47d36714-4e59-4a7e-a853-59f2a40c4fdc\") " Oct 30 19:17:56.948554 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.948530 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47d36714-4e59-4a7e-a853-59f2a40c4fdc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47d36714-4e59-4a7e-a853-59f2a40c4fdc-maas-api-openshift-ca\") pod \"47d36714-4e59-4a7e-a853-59f2a40c4fdc\" (UID: \"47d36714-4e59-4a7e-a853-59f2a40c4fdc\") " Oct 30 19:17:56.948770 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.948557 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47d36714-4e59-4a7e-a853-59f2a40c4fdc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47d36714-4e59-4a7e-a853-59f2a40c4fdc-maas-api-xks-ca\") pod \"47d36714-4e59-4a7e-a853-59f2a40c4fdc\" (UID: \"47d36714-4e59-4a7e-a853-59f2a40c4fdc\") " Oct 30 19:17:56.948909 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.948884 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d36714-4e59-4a7e-a853-59f2a40c4fdc-maas-api-openshift-ca" pod "47d36714-4e59-4a7e-a853-59f2a40c4fdc" (UID: "47d36714-4e59-4a7e-a853-59f2a40c4fdc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:56.951156 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.951086 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47d36714-4e59-4a7e-a853-59f2a40c4fdc-maas-api-xks-ca" pod "47d36714-4e59-4a7e-a853-59f2a40c4fdc" (UID: "47d36714-4e59-4a7e-a853-59f2a40c4fdc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:56.951287 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:56.951230 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d36714-4e59-4a7e-a853-59f2a40c4fdc-kube-api-access-xpxfx" pod "47d36714-4e59-4a7e-a853-59f2a40c4fdc" (UID: "47d36714-4e59-4a7e-a853-59f2a40c4fdc"). InnerVolumeSpecName "kube-api-access-xpxfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:57.049958 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:57.049920 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpxfx\" (UniqueName: \"kubernetes.io/projected/47d36714-4e59-4a7e-a853-59f2a40c4fdc-kube-api-access-xpxfx\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:57.049958 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:57.049950 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47d36714-4e59-4a7e-a853-59f2a40c4fdc-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:57.049958 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:57.049964 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47d36714-4e59-4a7e-a853-59f2a40c4fdc-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:17:57.648481 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:57.648439 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" event={"ID":"b5ecb161-fdf5-4084-9080-0227a442d88a","Type":"ContainerStarted","Data":"e095ac284e2cafb5255bfacf6d35d9c2f4431284b0b3d5d1d2dbbc5dcaf5daf2"} Oct 30 19:17:57.648684 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:57.648509 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" event={"ID":"b5ecb161-fdf5-4084-9080-0227a442d88a","Type":"ContainerStarted","Data":"85e27ec20a9899838da673e3b6e7b09e0b7883b2a245d56b3eaa2df9aa62d4a8"} Oct 30 19:17:57.649853 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:57.649826 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca9728-jtdjp" event={"ID":"47d36714-4e59-4a7e-a853-59f2a40c4fdc","Type":"ContainerDied","Data":"a8bc34d7959e3274f34e871c93affeabfe133557e2a454ab5f7fa401e279ebd4"} Oct 30 19:17:57.649988 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:57.649865 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8bc34d7959e3274f34e871c93affeabfe133557e2a454ab5f7fa401e279ebd4" Oct 30 19:17:57.649988 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:57.649837 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cca9728-jtdjp" Oct 30 19:17:57.842234 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:57.840620 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" podStartSLOduration=2.840598243 podStartE2EDuration="2.840598243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:57.821658232 +0000 UTC m=+2637.848763192" watchObservedRunningTime="2026-10-30 19:17:57.840598243 +0000 UTC m=+2637.867703203" Oct 30 19:17:58.266057 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:58.266019 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:17:58.267378 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:58.267354 4531 patch_prober.go:29] interesting pod/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 19:17:58.267465 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:58.267403 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" podUID="b5ecb161-fdf5-4084-9080-0227a442d88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 19:17:59.265786 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:59.265749 4531 patch_prober.go:29] interesting pod/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 19:17:59.266321 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:17:59.265811 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" podUID="b5ecb161-fdf5-4084-9080-0227a442d88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 19:18:00.265884 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:00.265846 4531 patch_prober.go:29] interesting pod/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 19:18:00.266341 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:00.265907 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" podUID="b5ecb161-fdf5-4084-9080-0227a442d88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 19:18:01.265877 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:01.265841 4531 patch_prober.go:29] interesting pod/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 19:18:01.266114 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:01.265903 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" podUID="b5ecb161-fdf5-4084-9080-0227a442d88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 19:18:02.265624 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:02.265587 4531 patch_prober.go:29] interesting pod/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 19:18:02.265822 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:02.265652 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" podUID="b5ecb161-fdf5-4084-9080-0227a442d88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 19:18:02.821752 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:02.821714 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-253a5641-2fn7r"] Oct 30 19:18:02.822237 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:02.822221 4531 cpu_manager.go:414] "containerMap: removing container" podUID="47d36714-4e59-4a7e-a853-59f2a40c4fdc" containerName="revoke-keys" Oct 30 19:18:02.822301 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:02.822237 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47d36714-4e59-4a7e-a853-59f2a40c4fdc" containerName="revoke-keys" Oct 30 19:18:02.822365 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:02.822354 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="47d36714-4e59-4a7e-a853-59f2a40c4fdc" containerName="revoke-keys" Oct 30 19:18:02.826962 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:02.826934 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5641-2fn7r" Oct 30 19:18:02.886183 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:02.886148 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:18:02.886417 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:02.886368 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:18:02.912820 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:02.912776 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d83c04f6-4c26-4b1f-916a-7ebfee306cbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-253a5641-2fn7r\" (UID: \"d83c04f6-4c26-4b1f-916a-7ebfee306cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5641-2fn7r" Oct 30 19:18:02.913015 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:02.912919 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlwln\" (UniqueName: \"kubernetes.io/projected/d83c04f6-4c26-4b1f-916a-7ebfee306cbd-kube-api-access-qlwln\") pod \"maas-api-revoke-sub-253a5641-2fn7r\" (UID: \"d83c04f6-4c26-4b1f-916a-7ebfee306cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5641-2fn7r" Oct 30 19:18:02.913015 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:02.912992 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d83c04f6-4c26-4b1f-916a-7ebfee306cbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-253a5641-2fn7r\" (UID: \"d83c04f6-4c26-4b1f-916a-7ebfee306cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5641-2fn7r" Oct 30 19:18:02.935679 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:02.935643 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:18:02.935895 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:02.935809 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:18:02.939047 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:02.939015 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-253a5641-2fn7r"] Oct 30 19:18:03.013990 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:03.013950 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d83c04f6-4c26-4b1f-916a-7ebfee306cbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-253a5641-2fn7r\" (UID: \"d83c04f6-4c26-4b1f-916a-7ebfee306cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5641-2fn7r" Oct 30 19:18:03.014168 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:03.014025 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlwln\" (UniqueName: \"kubernetes.io/projected/d83c04f6-4c26-4b1f-916a-7ebfee306cbd-kube-api-access-qlwln\") pod \"maas-api-revoke-sub-253a5641-2fn7r\" (UID: \"d83c04f6-4c26-4b1f-916a-7ebfee306cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5641-2fn7r" Oct 30 19:18:03.014168 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:03.014071 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d83c04f6-4c26-4b1f-916a-7ebfee306cbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-253a5641-2fn7r\" (UID: \"d83c04f6-4c26-4b1f-916a-7ebfee306cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5641-2fn7r" Oct 30 19:18:03.014711 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:03.014682 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d83c04f6-4c26-4b1f-916a-7ebfee306cbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-253a5641-2fn7r\" (UID: \"d83c04f6-4c26-4b1f-916a-7ebfee306cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5641-2fn7r" Oct 30 19:18:03.016854 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:03.016820 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d83c04f6-4c26-4b1f-916a-7ebfee306cbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-253a5641-2fn7r\" (UID: \"d83c04f6-4c26-4b1f-916a-7ebfee306cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5641-2fn7r" Oct 30 19:18:03.076062 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:03.075971 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlwln\" (UniqueName: \"kubernetes.io/projected/d83c04f6-4c26-4b1f-916a-7ebfee306cbd-kube-api-access-qlwln\") pod \"maas-api-revoke-sub-253a5641-2fn7r\" (UID: \"d83c04f6-4c26-4b1f-916a-7ebfee306cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5641-2fn7r" Oct 30 19:18:03.140852 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:03.140818 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5641-2fn7r" Oct 30 19:18:03.265817 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:03.265778 4531 patch_prober.go:29] interesting pod/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 19:18:03.266010 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:03.265844 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" podUID="b5ecb161-fdf5-4084-9080-0227a442d88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 19:18:03.530912 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:03.530875 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-253a5641-2fn7r"] Oct 30 19:18:03.531035 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:18:03.530885 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd83c04f6_4c26_4b1f_916a_7ebfee306cbd.slice/crio-09819642b247e334b5588eadf196c671694ba9dfa76cdc6cbb2b86b7e6060a6c WatchSource:0}: Error finding container 09819642b247e334b5588eadf196c671694ba9dfa76cdc6cbb2b86b7e6060a6c: Status 404 returned error can't find the container with id 09819642b247e334b5588eadf196c671694ba9dfa76cdc6cbb2b86b7e6060a6c Oct 30 19:18:03.698062 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:03.698026 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5641-2fn7r" event={"ID":"d83c04f6-4c26-4b1f-916a-7ebfee306cbd","Type":"ContainerStarted","Data":"09819642b247e334b5588eadf196c671694ba9dfa76cdc6cbb2b86b7e6060a6c"} Oct 30 19:18:04.024659 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:04.024613 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw"] Oct 30 19:18:04.028726 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:04.028698 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw" Oct 30 19:18:04.089613 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:04.089572 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw"] Oct 30 19:18:04.128717 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:04.128677 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j59qq\" (UniqueName: \"kubernetes.io/projected/de0f819c-c2c3-4022-afee-425bd1a73c69-kube-api-access-j59qq\") pod \"maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw\" (UID: \"de0f819c-c2c3-4022-afee-425bd1a73c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw" Oct 30 19:18:04.128883 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:04.128745 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de0f819c-c2c3-4022-afee-425bd1a73c69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw\" (UID: \"de0f819c-c2c3-4022-afee-425bd1a73c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw" Oct 30 19:18:04.128947 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:04.128891 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de0f819c-c2c3-4022-afee-425bd1a73c69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw\" (UID: \"de0f819c-c2c3-4022-afee-425bd1a73c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw" Oct 30 19:18:04.230091 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:04.229987 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de0f819c-c2c3-4022-afee-425bd1a73c69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw\" (UID: \"de0f819c-c2c3-4022-afee-425bd1a73c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw" Oct 30 19:18:04.230315 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:04.230123 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j59qq\" (UniqueName: \"kubernetes.io/projected/de0f819c-c2c3-4022-afee-425bd1a73c69-kube-api-access-j59qq\") pod \"maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw\" (UID: \"de0f819c-c2c3-4022-afee-425bd1a73c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw" Oct 30 19:18:04.230315 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:04.230204 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de0f819c-c2c3-4022-afee-425bd1a73c69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw\" (UID: \"de0f819c-c2c3-4022-afee-425bd1a73c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw" Oct 30 19:18:04.230783 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:04.230762 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de0f819c-c2c3-4022-afee-425bd1a73c69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw\" (UID: \"de0f819c-c2c3-4022-afee-425bd1a73c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw" Oct 30 19:18:04.232641 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:04.232618 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de0f819c-c2c3-4022-afee-425bd1a73c69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw\" (UID: \"de0f819c-c2c3-4022-afee-425bd1a73c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw" Oct 30 19:18:04.270887 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:04.270854 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:18:04.271198 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:04.271162 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:18:04.271987 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:04.271971 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:18:04.293472 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:04.293431 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j59qq\" (UniqueName: \"kubernetes.io/projected/de0f819c-c2c3-4022-afee-425bd1a73c69-kube-api-access-j59qq\") pod \"maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw\" (UID: \"de0f819c-c2c3-4022-afee-425bd1a73c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw" Oct 30 19:18:04.348716 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:04.348669 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw" Oct 30 19:18:04.707278 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:04.707244 4531 generic.go:356] "Generic (PLEG): container finished" podID="d83c04f6-4c26-4b1f-916a-7ebfee306cbd" containerID="7004cd34afdbea6ebb9f865228f416c52366b3d037963288c98ea2aac60b3026" exitCode=0 Oct 30 19:18:04.707502 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:04.707322 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5641-2fn7r" event={"ID":"d83c04f6-4c26-4b1f-916a-7ebfee306cbd","Type":"ContainerDied","Data":"7004cd34afdbea6ebb9f865228f416c52366b3d037963288c98ea2aac60b3026"} Oct 30 19:18:04.739180 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:04.739156 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw"] Oct 30 19:18:04.740737 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:18:04.740712 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde0f819c_c2c3_4022_afee_425bd1a73c69.slice/crio-85e69987a2551508eaf421e79a6836eea260cee483c8c073768889023dea1405 WatchSource:0}: Error finding container 85e69987a2551508eaf421e79a6836eea260cee483c8c073768889023dea1405: Status 404 returned error can't find the container with id 85e69987a2551508eaf421e79a6836eea260cee483c8c073768889023dea1405 Oct 30 19:18:05.732484 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:05.732439 4531 generic.go:356] "Generic (PLEG): container finished" podID="de0f819c-c2c3-4022-afee-425bd1a73c69" containerID="b9cdbe3269ba6252b6d7f236dd95fb6a8017b2b112a4499432f61aa86bf0442c" exitCode=0 Oct 30 19:18:05.732991 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:05.732530 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw" event={"ID":"de0f819c-c2c3-4022-afee-425bd1a73c69","Type":"ContainerDied","Data":"b9cdbe3269ba6252b6d7f236dd95fb6a8017b2b112a4499432f61aa86bf0442c"} Oct 30 19:18:05.732991 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:05.732634 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw" event={"ID":"de0f819c-c2c3-4022-afee-425bd1a73c69","Type":"ContainerStarted","Data":"85e69987a2551508eaf421e79a6836eea260cee483c8c073768889023dea1405"} Oct 30 19:18:05.908909 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:05.908872 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5641-2fn7r" Oct 30 19:18:05.953103 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:05.952988 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d83c04f6-4c26-4b1f-916a-7ebfee306cbd-kube-api-access-qlwln\" (UniqueName: \"kubernetes.io/projected/d83c04f6-4c26-4b1f-916a-7ebfee306cbd-kube-api-access-qlwln\") pod \"d83c04f6-4c26-4b1f-916a-7ebfee306cbd\" (UID: \"d83c04f6-4c26-4b1f-916a-7ebfee306cbd\") " Oct 30 19:18:05.953312 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:05.953161 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d83c04f6-4c26-4b1f-916a-7ebfee306cbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d83c04f6-4c26-4b1f-916a-7ebfee306cbd-maas-api-xks-ca\") pod \"d83c04f6-4c26-4b1f-916a-7ebfee306cbd\" (UID: \"d83c04f6-4c26-4b1f-916a-7ebfee306cbd\") " Oct 30 19:18:05.953312 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:05.953245 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d83c04f6-4c26-4b1f-916a-7ebfee306cbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d83c04f6-4c26-4b1f-916a-7ebfee306cbd-maas-api-openshift-ca\") pod \"d83c04f6-4c26-4b1f-916a-7ebfee306cbd\" (UID: \"d83c04f6-4c26-4b1f-916a-7ebfee306cbd\") " Oct 30 19:18:05.953674 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:05.953631 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d83c04f6-4c26-4b1f-916a-7ebfee306cbd-maas-api-openshift-ca" pod "d83c04f6-4c26-4b1f-916a-7ebfee306cbd" (UID: "d83c04f6-4c26-4b1f-916a-7ebfee306cbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:18:05.955629 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:05.955597 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d83c04f6-4c26-4b1f-916a-7ebfee306cbd-maas-api-xks-ca" pod "d83c04f6-4c26-4b1f-916a-7ebfee306cbd" (UID: "d83c04f6-4c26-4b1f-916a-7ebfee306cbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:18:05.955975 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:05.955948 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d83c04f6-4c26-4b1f-916a-7ebfee306cbd-kube-api-access-qlwln" pod "d83c04f6-4c26-4b1f-916a-7ebfee306cbd" (UID: "d83c04f6-4c26-4b1f-916a-7ebfee306cbd"). InnerVolumeSpecName "kube-api-access-qlwln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:18:06.054256 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:06.054219 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d83c04f6-4c26-4b1f-916a-7ebfee306cbd-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:06.054256 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:06.054256 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d83c04f6-4c26-4b1f-916a-7ebfee306cbd-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:06.054256 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:06.054270 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlwln\" (UniqueName: \"kubernetes.io/projected/d83c04f6-4c26-4b1f-916a-7ebfee306cbd-kube-api-access-qlwln\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:06.741546 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:06.741512 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5641-2fn7r" Oct 30 19:18:06.741546 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:06.741541 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5641-2fn7r" event={"ID":"d83c04f6-4c26-4b1f-916a-7ebfee306cbd","Type":"ContainerDied","Data":"09819642b247e334b5588eadf196c671694ba9dfa76cdc6cbb2b86b7e6060a6c"} Oct 30 19:18:06.742084 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:06.741594 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09819642b247e334b5588eadf196c671694ba9dfa76cdc6cbb2b86b7e6060a6c" Oct 30 19:18:06.916276 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:06.916242 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw" Oct 30 19:18:06.964227 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:06.964176 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de0f819c-c2c3-4022-afee-425bd1a73c69-kube-api-access-j59qq\" (UniqueName: \"kubernetes.io/projected/de0f819c-c2c3-4022-afee-425bd1a73c69-kube-api-access-j59qq\") pod \"de0f819c-c2c3-4022-afee-425bd1a73c69\" (UID: \"de0f819c-c2c3-4022-afee-425bd1a73c69\") " Oct 30 19:18:06.964440 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:06.964306 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de0f819c-c2c3-4022-afee-425bd1a73c69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de0f819c-c2c3-4022-afee-425bd1a73c69-maas-api-xks-ca\") pod \"de0f819c-c2c3-4022-afee-425bd1a73c69\" (UID: \"de0f819c-c2c3-4022-afee-425bd1a73c69\") " Oct 30 19:18:06.964440 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:06.964379 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de0f819c-c2c3-4022-afee-425bd1a73c69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de0f819c-c2c3-4022-afee-425bd1a73c69-maas-api-openshift-ca\") pod \"de0f819c-c2c3-4022-afee-425bd1a73c69\" (UID: \"de0f819c-c2c3-4022-afee-425bd1a73c69\") " Oct 30 19:18:06.964767 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:06.964738 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de0f819c-c2c3-4022-afee-425bd1a73c69-maas-api-openshift-ca" pod "de0f819c-c2c3-4022-afee-425bd1a73c69" (UID: "de0f819c-c2c3-4022-afee-425bd1a73c69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:18:06.964909 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:06.964885 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de0f819c-c2c3-4022-afee-425bd1a73c69-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:06.966662 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:06.966637 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de0f819c-c2c3-4022-afee-425bd1a73c69-maas-api-xks-ca" pod "de0f819c-c2c3-4022-afee-425bd1a73c69" (UID: "de0f819c-c2c3-4022-afee-425bd1a73c69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:18:06.966760 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:06.966659 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de0f819c-c2c3-4022-afee-425bd1a73c69-kube-api-access-j59qq" pod "de0f819c-c2c3-4022-afee-425bd1a73c69" (UID: "de0f819c-c2c3-4022-afee-425bd1a73c69"). InnerVolumeSpecName "kube-api-access-j59qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:18:07.065750 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:07.065708 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j59qq\" (UniqueName: \"kubernetes.io/projected/de0f819c-c2c3-4022-afee-425bd1a73c69-kube-api-access-j59qq\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:07.065750 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:07.065752 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de0f819c-c2c3-4022-afee-425bd1a73c69-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:07.749606 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:07.749571 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw" Oct 30 19:18:07.749989 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:07.749611 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw" event={"ID":"de0f819c-c2c3-4022-afee-425bd1a73c69","Type":"ContainerDied","Data":"85e69987a2551508eaf421e79a6836eea260cee483c8c073768889023dea1405"} Oct 30 19:18:07.749989 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:07.749663 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85e69987a2551508eaf421e79a6836eea260cee483c8c073768889023dea1405" Oct 30 19:18:09.068571 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.068535 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70c1191d-wlzjh"] Oct 30 19:18:09.068992 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.068979 4531 cpu_manager.go:414] "containerMap: removing container" podUID="d83c04f6-4c26-4b1f-916a-7ebfee306cbd" containerName="revoke-keys" Oct 30 19:18:09.069031 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.068993 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d83c04f6-4c26-4b1f-916a-7ebfee306cbd" containerName="revoke-keys" Oct 30 19:18:09.069031 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.069004 4531 cpu_manager.go:414] "containerMap: removing container" podUID="de0f819c-c2c3-4022-afee-425bd1a73c69" containerName="revoke-keys" Oct 30 19:18:09.069031 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.069011 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de0f819c-c2c3-4022-afee-425bd1a73c69" containerName="revoke-keys" Oct 30 19:18:09.069153 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.069079 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d83c04f6-4c26-4b1f-916a-7ebfee306cbd" containerName="revoke-keys" Oct 30 19:18:09.069153 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.069087 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="de0f819c-c2c3-4022-afee-425bd1a73c69" containerName="revoke-keys" Oct 30 19:18:09.072464 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.072447 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c1191d-wlzjh" Oct 30 19:18:09.090564 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.090541 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/106dc01f-378d-4b71-8da7-dbae2d87a0a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70c1191d-wlzjh\" (UID: \"106dc01f-378d-4b71-8da7-dbae2d87a0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c1191d-wlzjh" Oct 30 19:18:09.090694 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.090676 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr7m7\" (UniqueName: \"kubernetes.io/projected/106dc01f-378d-4b71-8da7-dbae2d87a0a8-kube-api-access-fr7m7\") pod \"maas-api-revoke-sub-70c1191d-wlzjh\" (UID: \"106dc01f-378d-4b71-8da7-dbae2d87a0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c1191d-wlzjh" Oct 30 19:18:09.090741 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.090726 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/106dc01f-378d-4b71-8da7-dbae2d87a0a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70c1191d-wlzjh\" (UID: \"106dc01f-378d-4b71-8da7-dbae2d87a0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c1191d-wlzjh" Oct 30 19:18:09.173743 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.173701 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:18:09.173743 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.173719 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:18:09.173963 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.173766 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:18:09.173963 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.173812 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:18:09.178321 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.178301 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70c1191d-wlzjh"] Oct 30 19:18:09.198203 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.198166 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr7m7\" (UniqueName: \"kubernetes.io/projected/106dc01f-378d-4b71-8da7-dbae2d87a0a8-kube-api-access-fr7m7\") pod \"maas-api-revoke-sub-70c1191d-wlzjh\" (UID: \"106dc01f-378d-4b71-8da7-dbae2d87a0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c1191d-wlzjh" Oct 30 19:18:09.198339 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.198214 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/106dc01f-378d-4b71-8da7-dbae2d87a0a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70c1191d-wlzjh\" (UID: \"106dc01f-378d-4b71-8da7-dbae2d87a0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c1191d-wlzjh" Oct 30 19:18:09.198339 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.198311 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/106dc01f-378d-4b71-8da7-dbae2d87a0a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70c1191d-wlzjh\" (UID: \"106dc01f-378d-4b71-8da7-dbae2d87a0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c1191d-wlzjh" Oct 30 19:18:09.198893 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.198868 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/106dc01f-378d-4b71-8da7-dbae2d87a0a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70c1191d-wlzjh\" (UID: \"106dc01f-378d-4b71-8da7-dbae2d87a0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c1191d-wlzjh" Oct 30 19:18:09.200844 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.200824 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/106dc01f-378d-4b71-8da7-dbae2d87a0a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70c1191d-wlzjh\" (UID: \"106dc01f-378d-4b71-8da7-dbae2d87a0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c1191d-wlzjh" Oct 30 19:18:09.259157 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.259098 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr7m7\" (UniqueName: \"kubernetes.io/projected/106dc01f-378d-4b71-8da7-dbae2d87a0a8-kube-api-access-fr7m7\") pod \"maas-api-revoke-sub-70c1191d-wlzjh\" (UID: \"106dc01f-378d-4b71-8da7-dbae2d87a0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c1191d-wlzjh" Oct 30 19:18:09.385007 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.384931 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c1191d-wlzjh" Oct 30 19:18:09.586272 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.586237 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a21902d6-npxp7"] Oct 30 19:18:09.590952 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.590929 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21902d6-npxp7" Oct 30 19:18:09.601716 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.601694 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6jnn\" (UniqueName: \"kubernetes.io/projected/ef8bd865-8f5f-4bb0-aa15-69d8cd946642-kube-api-access-t6jnn\") pod \"maas-api-revoke-sub-a21902d6-npxp7\" (UID: \"ef8bd865-8f5f-4bb0-aa15-69d8cd946642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21902d6-npxp7" Oct 30 19:18:09.601826 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.601792 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef8bd865-8f5f-4bb0-aa15-69d8cd946642-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a21902d6-npxp7\" (UID: \"ef8bd865-8f5f-4bb0-aa15-69d8cd946642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21902d6-npxp7" Oct 30 19:18:09.601896 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.601876 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef8bd865-8f5f-4bb0-aa15-69d8cd946642-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a21902d6-npxp7\" (UID: \"ef8bd865-8f5f-4bb0-aa15-69d8cd946642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21902d6-npxp7" Oct 30 19:18:09.649470 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.649404 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a21902d6-npxp7"] Oct 30 19:18:09.702785 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.702759 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6jnn\" (UniqueName: \"kubernetes.io/projected/ef8bd865-8f5f-4bb0-aa15-69d8cd946642-kube-api-access-t6jnn\") pod \"maas-api-revoke-sub-a21902d6-npxp7\" (UID: \"ef8bd865-8f5f-4bb0-aa15-69d8cd946642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21902d6-npxp7" Oct 30 19:18:09.702947 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.702836 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef8bd865-8f5f-4bb0-aa15-69d8cd946642-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a21902d6-npxp7\" (UID: \"ef8bd865-8f5f-4bb0-aa15-69d8cd946642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21902d6-npxp7" Oct 30 19:18:09.702947 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.702883 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef8bd865-8f5f-4bb0-aa15-69d8cd946642-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a21902d6-npxp7\" (UID: \"ef8bd865-8f5f-4bb0-aa15-69d8cd946642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21902d6-npxp7" Oct 30 19:18:09.703671 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.703642 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef8bd865-8f5f-4bb0-aa15-69d8cd946642-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a21902d6-npxp7\" (UID: \"ef8bd865-8f5f-4bb0-aa15-69d8cd946642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21902d6-npxp7" Oct 30 19:18:09.705516 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.705492 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef8bd865-8f5f-4bb0-aa15-69d8cd946642-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a21902d6-npxp7\" (UID: \"ef8bd865-8f5f-4bb0-aa15-69d8cd946642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21902d6-npxp7" Oct 30 19:18:09.760985 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.760956 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70c1191d-wlzjh"] Oct 30 19:18:09.763408 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:18:09.763369 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106dc01f_378d_4b71_8da7_dbae2d87a0a8.slice/crio-614e1e049b7b1f002cf4830efcf0e3664ffe0987763f84cf9df23d1ef487d13b WatchSource:0}: Error finding container 614e1e049b7b1f002cf4830efcf0e3664ffe0987763f84cf9df23d1ef487d13b: Status 404 returned error can't find the container with id 614e1e049b7b1f002cf4830efcf0e3664ffe0987763f84cf9df23d1ef487d13b Oct 30 19:18:09.765595 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.765572 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6jnn\" (UniqueName: \"kubernetes.io/projected/ef8bd865-8f5f-4bb0-aa15-69d8cd946642-kube-api-access-t6jnn\") pod \"maas-api-revoke-sub-a21902d6-npxp7\" (UID: \"ef8bd865-8f5f-4bb0-aa15-69d8cd946642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21902d6-npxp7" Oct 30 19:18:09.904873 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:09.904798 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21902d6-npxp7" Oct 30 19:18:10.283389 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:10.283362 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a21902d6-npxp7"] Oct 30 19:18:10.284861 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:18:10.284837 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8bd865_8f5f_4bb0_aa15_69d8cd946642.slice/crio-89458315259f7a80769248fa80b2527aab46ff0913bce48408b62c7c47cc7bad WatchSource:0}: Error finding container 89458315259f7a80769248fa80b2527aab46ff0913bce48408b62c7c47cc7bad: Status 404 returned error can't find the container with id 89458315259f7a80769248fa80b2527aab46ff0913bce48408b62c7c47cc7bad Oct 30 19:18:10.771698 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:10.771663 4531 generic.go:356] "Generic (PLEG): container finished" podID="106dc01f-378d-4b71-8da7-dbae2d87a0a8" containerID="d81766ce9b60f917810aa717508324a0d3f9bfdb99f4f2d5aff0896bda3663d6" exitCode=0 Oct 30 19:18:10.771941 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:10.771759 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c1191d-wlzjh" event={"ID":"106dc01f-378d-4b71-8da7-dbae2d87a0a8","Type":"ContainerDied","Data":"d81766ce9b60f917810aa717508324a0d3f9bfdb99f4f2d5aff0896bda3663d6"} Oct 30 19:18:10.771941 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:10.771824 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c1191d-wlzjh" event={"ID":"106dc01f-378d-4b71-8da7-dbae2d87a0a8","Type":"ContainerStarted","Data":"614e1e049b7b1f002cf4830efcf0e3664ffe0987763f84cf9df23d1ef487d13b"} Oct 30 19:18:10.773385 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:10.773355 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21902d6-npxp7" event={"ID":"ef8bd865-8f5f-4bb0-aa15-69d8cd946642","Type":"ContainerStarted","Data":"8df31c6fe6cc732006de2c604233a83d229105a1103297275537500dc35d5c16"} Oct 30 19:18:10.773499 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:10.773403 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21902d6-npxp7" event={"ID":"ef8bd865-8f5f-4bb0-aa15-69d8cd946642","Type":"ContainerStarted","Data":"89458315259f7a80769248fa80b2527aab46ff0913bce48408b62c7c47cc7bad"} Oct 30 19:18:11.009945 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:11.009892 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21902d6-npxp7" podStartSLOduration=2.009877787 podStartE2EDuration="2.009877787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:11.006883942 +0000 UTC m=+2651.033988905" watchObservedRunningTime="2026-10-30 19:18:11.009877787 +0000 UTC m=+2651.036982746" Oct 30 19:18:11.781409 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:11.781363 4531 generic.go:356] "Generic (PLEG): container finished" podID="ef8bd865-8f5f-4bb0-aa15-69d8cd946642" containerID="8df31c6fe6cc732006de2c604233a83d229105a1103297275537500dc35d5c16" exitCode=0 Oct 30 19:18:11.781882 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:11.781452 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21902d6-npxp7" event={"ID":"ef8bd865-8f5f-4bb0-aa15-69d8cd946642","Type":"ContainerDied","Data":"8df31c6fe6cc732006de2c604233a83d229105a1103297275537500dc35d5c16"} Oct 30 19:18:11.928896 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:11.928869 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c1191d-wlzjh" Oct 30 19:18:12.023789 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:12.023746 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/106dc01f-378d-4b71-8da7-dbae2d87a0a8-kube-api-access-fr7m7\" (UniqueName: \"kubernetes.io/projected/106dc01f-378d-4b71-8da7-dbae2d87a0a8-kube-api-access-fr7m7\") pod \"106dc01f-378d-4b71-8da7-dbae2d87a0a8\" (UID: \"106dc01f-378d-4b71-8da7-dbae2d87a0a8\") " Oct 30 19:18:12.023950 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:12.023908 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/106dc01f-378d-4b71-8da7-dbae2d87a0a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/106dc01f-378d-4b71-8da7-dbae2d87a0a8-maas-api-openshift-ca\") pod \"106dc01f-378d-4b71-8da7-dbae2d87a0a8\" (UID: \"106dc01f-378d-4b71-8da7-dbae2d87a0a8\") " Oct 30 19:18:12.023950 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:12.023941 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/106dc01f-378d-4b71-8da7-dbae2d87a0a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/106dc01f-378d-4b71-8da7-dbae2d87a0a8-maas-api-xks-ca\") pod \"106dc01f-378d-4b71-8da7-dbae2d87a0a8\" (UID: \"106dc01f-378d-4b71-8da7-dbae2d87a0a8\") " Oct 30 19:18:12.024351 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:12.024321 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/106dc01f-378d-4b71-8da7-dbae2d87a0a8-maas-api-openshift-ca" pod "106dc01f-378d-4b71-8da7-dbae2d87a0a8" (UID: "106dc01f-378d-4b71-8da7-dbae2d87a0a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:18:12.025986 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:12.025954 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/106dc01f-378d-4b71-8da7-dbae2d87a0a8-kube-api-access-fr7m7" pod "106dc01f-378d-4b71-8da7-dbae2d87a0a8" (UID: "106dc01f-378d-4b71-8da7-dbae2d87a0a8"). InnerVolumeSpecName "kube-api-access-fr7m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:18:12.026094 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:12.026049 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/106dc01f-378d-4b71-8da7-dbae2d87a0a8-maas-api-xks-ca" pod "106dc01f-378d-4b71-8da7-dbae2d87a0a8" (UID: "106dc01f-378d-4b71-8da7-dbae2d87a0a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:18:12.125423 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:12.125378 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/106dc01f-378d-4b71-8da7-dbae2d87a0a8-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:12.125423 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:12.125413 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/106dc01f-378d-4b71-8da7-dbae2d87a0a8-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:12.125624 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:12.125427 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr7m7\" (UniqueName: \"kubernetes.io/projected/106dc01f-378d-4b71-8da7-dbae2d87a0a8-kube-api-access-fr7m7\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:12.789386 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:12.789348 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c1191d-wlzjh" Oct 30 19:18:12.789825 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:12.789400 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c1191d-wlzjh" event={"ID":"106dc01f-378d-4b71-8da7-dbae2d87a0a8","Type":"ContainerDied","Data":"614e1e049b7b1f002cf4830efcf0e3664ffe0987763f84cf9df23d1ef487d13b"} Oct 30 19:18:12.789825 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:12.789457 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="614e1e049b7b1f002cf4830efcf0e3664ffe0987763f84cf9df23d1ef487d13b" Oct 30 19:18:12.942729 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:12.942699 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21902d6-npxp7" Oct 30 19:18:13.035408 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:13.035369 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef8bd865-8f5f-4bb0-aa15-69d8cd946642-kube-api-access-t6jnn\" (UniqueName: \"kubernetes.io/projected/ef8bd865-8f5f-4bb0-aa15-69d8cd946642-kube-api-access-t6jnn\") pod \"ef8bd865-8f5f-4bb0-aa15-69d8cd946642\" (UID: \"ef8bd865-8f5f-4bb0-aa15-69d8cd946642\") " Oct 30 19:18:13.035635 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:13.035442 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef8bd865-8f5f-4bb0-aa15-69d8cd946642-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef8bd865-8f5f-4bb0-aa15-69d8cd946642-maas-api-openshift-ca\") pod \"ef8bd865-8f5f-4bb0-aa15-69d8cd946642\" (UID: \"ef8bd865-8f5f-4bb0-aa15-69d8cd946642\") " Oct 30 19:18:13.035635 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:13.035464 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef8bd865-8f5f-4bb0-aa15-69d8cd946642-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef8bd865-8f5f-4bb0-aa15-69d8cd946642-maas-api-xks-ca\") pod \"ef8bd865-8f5f-4bb0-aa15-69d8cd946642\" (UID: \"ef8bd865-8f5f-4bb0-aa15-69d8cd946642\") " Oct 30 19:18:13.035811 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:13.035784 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef8bd865-8f5f-4bb0-aa15-69d8cd946642-maas-api-openshift-ca" pod "ef8bd865-8f5f-4bb0-aa15-69d8cd946642" (UID: "ef8bd865-8f5f-4bb0-aa15-69d8cd946642"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:18:13.035891 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:13.035877 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef8bd865-8f5f-4bb0-aa15-69d8cd946642-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:13.037764 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:13.037740 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef8bd865-8f5f-4bb0-aa15-69d8cd946642-maas-api-xks-ca" pod "ef8bd865-8f5f-4bb0-aa15-69d8cd946642" (UID: "ef8bd865-8f5f-4bb0-aa15-69d8cd946642"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:18:13.037858 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:13.037803 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef8bd865-8f5f-4bb0-aa15-69d8cd946642-kube-api-access-t6jnn" pod "ef8bd865-8f5f-4bb0-aa15-69d8cd946642" (UID: "ef8bd865-8f5f-4bb0-aa15-69d8cd946642"). InnerVolumeSpecName "kube-api-access-t6jnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:18:13.137276 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:13.137226 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6jnn\" (UniqueName: \"kubernetes.io/projected/ef8bd865-8f5f-4bb0-aa15-69d8cd946642-kube-api-access-t6jnn\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:13.137276 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:13.137261 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef8bd865-8f5f-4bb0-aa15-69d8cd946642-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:13.799080 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:13.799049 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21902d6-npxp7" Oct 30 19:18:13.799501 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:13.799073 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21902d6-npxp7" event={"ID":"ef8bd865-8f5f-4bb0-aa15-69d8cd946642","Type":"ContainerDied","Data":"89458315259f7a80769248fa80b2527aab46ff0913bce48408b62c7c47cc7bad"} Oct 30 19:18:13.799501 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:13.799145 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89458315259f7a80769248fa80b2527aab46ff0913bce48408b62c7c47cc7bad" Oct 30 19:18:16.258616 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:16.258575 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a24f4dc-25blv"] Oct 30 19:18:16.259212 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:16.259193 4531 cpu_manager.go:414] "containerMap: removing container" podUID="ef8bd865-8f5f-4bb0-aa15-69d8cd946642" containerName="revoke-keys" Oct 30 19:18:16.259265 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:16.259214 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef8bd865-8f5f-4bb0-aa15-69d8cd946642" containerName="revoke-keys" Oct 30 19:18:16.259265 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:16.259237 4531 cpu_manager.go:414] "containerMap: removing container" podUID="106dc01f-378d-4b71-8da7-dbae2d87a0a8" containerName="revoke-keys" Oct 30 19:18:16.259265 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:16.259245 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="106dc01f-378d-4b71-8da7-dbae2d87a0a8" containerName="revoke-keys" Oct 30 19:18:16.259391 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:16.259379 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef8bd865-8f5f-4bb0-aa15-69d8cd946642" containerName="revoke-keys" Oct 30 19:18:16.259429 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:16.259397 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="106dc01f-378d-4b71-8da7-dbae2d87a0a8" containerName="revoke-keys" Oct 30 19:18:16.263479 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:16.263457 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24f4dc-25blv" Oct 30 19:18:16.317286 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:16.317248 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:18:16.321751 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:16.321720 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a24f4dc-25blv"] Oct 30 19:18:16.366633 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:16.366601 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:18:16.366633 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:16.366601 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:18:16.366633 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:16.366607 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:18:16.372813 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:16.372784 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl562\" (UniqueName: \"kubernetes.io/projected/93a22968-0fc8-4713-b803-6c42a162192d-kube-api-access-dl562\") pod \"maas-api-revoke-sub-2a24f4dc-25blv\" (UID: \"93a22968-0fc8-4713-b803-6c42a162192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24f4dc-25blv" Oct 30 19:18:16.372969 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:16.372834 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93a22968-0fc8-4713-b803-6c42a162192d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a24f4dc-25blv\" (UID: \"93a22968-0fc8-4713-b803-6c42a162192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24f4dc-25blv" Oct 30 19:18:16.373040 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:16.372984 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93a22968-0fc8-4713-b803-6c42a162192d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a24f4dc-25blv\" (UID: \"93a22968-0fc8-4713-b803-6c42a162192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24f4dc-25blv" Oct 30 19:18:16.474433 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:16.474391 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93a22968-0fc8-4713-b803-6c42a162192d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a24f4dc-25blv\" (UID: \"93a22968-0fc8-4713-b803-6c42a162192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24f4dc-25blv" Oct 30 19:18:16.474641 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:16.474517 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl562\" (UniqueName: \"kubernetes.io/projected/93a22968-0fc8-4713-b803-6c42a162192d-kube-api-access-dl562\") pod \"maas-api-revoke-sub-2a24f4dc-25blv\" (UID: \"93a22968-0fc8-4713-b803-6c42a162192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24f4dc-25blv" Oct 30 19:18:16.474641 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:16.474539 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93a22968-0fc8-4713-b803-6c42a162192d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a24f4dc-25blv\" (UID: \"93a22968-0fc8-4713-b803-6c42a162192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24f4dc-25blv" Oct 30 19:18:16.475177 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:16.475155 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93a22968-0fc8-4713-b803-6c42a162192d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a24f4dc-25blv\" (UID: \"93a22968-0fc8-4713-b803-6c42a162192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24f4dc-25blv" Oct 30 19:18:16.476971 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:16.476947 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93a22968-0fc8-4713-b803-6c42a162192d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a24f4dc-25blv\" (UID: \"93a22968-0fc8-4713-b803-6c42a162192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24f4dc-25blv" Oct 30 19:18:16.535914 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:16.535838 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl562\" (UniqueName: \"kubernetes.io/projected/93a22968-0fc8-4713-b803-6c42a162192d-kube-api-access-dl562\") pod \"maas-api-revoke-sub-2a24f4dc-25blv\" (UID: \"93a22968-0fc8-4713-b803-6c42a162192d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24f4dc-25blv" Oct 30 19:18:16.576552 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:16.576512 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24f4dc-25blv" Oct 30 19:18:16.959542 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:16.959511 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a24f4dc-25blv"] Oct 30 19:18:16.960617 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:18:16.960589 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a22968_0fc8_4713_b803_6c42a162192d.slice/crio-ba97b8ccd83a6a1ddeaf1899208b4f2a06bfa2e4df9437f3b58bb1d58c5a9fa6 WatchSource:0}: Error finding container ba97b8ccd83a6a1ddeaf1899208b4f2a06bfa2e4df9437f3b58bb1d58c5a9fa6: Status 404 returned error can't find the container with id ba97b8ccd83a6a1ddeaf1899208b4f2a06bfa2e4df9437f3b58bb1d58c5a9fa6 Oct 30 19:18:17.826421 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:17.826383 4531 generic.go:356] "Generic (PLEG): container finished" podID="93a22968-0fc8-4713-b803-6c42a162192d" containerID="207cbefc64277798c8a9578ad4f2e2459e210c79209eca938f196e42c5b2eee1" exitCode=0 Oct 30 19:18:17.826866 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:17.826418 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24f4dc-25blv" event={"ID":"93a22968-0fc8-4713-b803-6c42a162192d","Type":"ContainerDied","Data":"207cbefc64277798c8a9578ad4f2e2459e210c79209eca938f196e42c5b2eee1"} Oct 30 19:18:17.826866 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:17.826500 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24f4dc-25blv" event={"ID":"93a22968-0fc8-4713-b803-6c42a162192d","Type":"ContainerStarted","Data":"ba97b8ccd83a6a1ddeaf1899208b4f2a06bfa2e4df9437f3b58bb1d58c5a9fa6"} Oct 30 19:18:18.985282 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:18.985247 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24f4dc-25blv" Oct 30 19:18:19.103208 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:19.103083 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93a22968-0fc8-4713-b803-6c42a162192d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93a22968-0fc8-4713-b803-6c42a162192d-maas-api-xks-ca\") pod \"93a22968-0fc8-4713-b803-6c42a162192d\" (UID: \"93a22968-0fc8-4713-b803-6c42a162192d\") " Oct 30 19:18:19.103208 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:19.103158 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93a22968-0fc8-4713-b803-6c42a162192d-kube-api-access-dl562\" (UniqueName: \"kubernetes.io/projected/93a22968-0fc8-4713-b803-6c42a162192d-kube-api-access-dl562\") pod \"93a22968-0fc8-4713-b803-6c42a162192d\" (UID: \"93a22968-0fc8-4713-b803-6c42a162192d\") " Oct 30 19:18:19.103877 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:19.103586 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93a22968-0fc8-4713-b803-6c42a162192d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93a22968-0fc8-4713-b803-6c42a162192d-maas-api-openshift-ca\") pod \"93a22968-0fc8-4713-b803-6c42a162192d\" (UID: \"93a22968-0fc8-4713-b803-6c42a162192d\") " Oct 30 19:18:19.104439 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:19.104410 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a22968-0fc8-4713-b803-6c42a162192d-maas-api-openshift-ca" pod "93a22968-0fc8-4713-b803-6c42a162192d" (UID: "93a22968-0fc8-4713-b803-6c42a162192d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:18:19.105906 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:19.105824 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a22968-0fc8-4713-b803-6c42a162192d-maas-api-xks-ca" pod "93a22968-0fc8-4713-b803-6c42a162192d" (UID: "93a22968-0fc8-4713-b803-6c42a162192d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:18:19.106079 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:19.106057 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a22968-0fc8-4713-b803-6c42a162192d-kube-api-access-dl562" pod "93a22968-0fc8-4713-b803-6c42a162192d" (UID: "93a22968-0fc8-4713-b803-6c42a162192d"). InnerVolumeSpecName "kube-api-access-dl562". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:18:19.205837 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:19.205789 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93a22968-0fc8-4713-b803-6c42a162192d-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:19.205837 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:19.205832 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl562\" (UniqueName: \"kubernetes.io/projected/93a22968-0fc8-4713-b803-6c42a162192d-kube-api-access-dl562\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:19.206069 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:19.205849 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93a22968-0fc8-4713-b803-6c42a162192d-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:19.841836 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:19.841806 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24f4dc-25blv" Oct 30 19:18:19.842040 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:19.841801 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24f4dc-25blv" event={"ID":"93a22968-0fc8-4713-b803-6c42a162192d","Type":"ContainerDied","Data":"ba97b8ccd83a6a1ddeaf1899208b4f2a06bfa2e4df9437f3b58bb1d58c5a9fa6"} Oct 30 19:18:19.842040 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:19.841943 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba97b8ccd83a6a1ddeaf1899208b4f2a06bfa2e4df9437f3b58bb1d58c5a9fa6" Oct 30 19:18:26.396640 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:18:26.396602 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:18:32.797369 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:32.797339 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92a026c3-prpbp"] Oct 30 19:18:32.797954 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:32.797934 4531 cpu_manager.go:414] "containerMap: removing container" podUID="93a22968-0fc8-4713-b803-6c42a162192d" containerName="revoke-keys" Oct 30 19:18:32.798058 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:32.797957 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a22968-0fc8-4713-b803-6c42a162192d" containerName="revoke-keys" Oct 30 19:18:32.798123 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:32.798112 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="93a22968-0fc8-4713-b803-6c42a162192d" containerName="revoke-keys" Oct 30 19:18:32.801481 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:32.801461 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a026c3-prpbp" Oct 30 19:18:32.857518 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:32.857490 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:18:32.906841 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:32.906815 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:18:32.906841 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:32.906815 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:18:32.907027 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:32.906841 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:18:32.910403 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:32.910380 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92a026c3-prpbp"] Oct 30 19:18:32.990444 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:32.990411 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkkqg\" (UniqueName: \"kubernetes.io/projected/4976184c-ddb4-4a68-8af2-27e0a316a1df-kube-api-access-tkkqg\") pod \"maas-api-revoke-sub-92a026c3-prpbp\" (UID: \"4976184c-ddb4-4a68-8af2-27e0a316a1df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a026c3-prpbp" Oct 30 19:18:32.990444 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:32.990444 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4976184c-ddb4-4a68-8af2-27e0a316a1df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92a026c3-prpbp\" (UID: \"4976184c-ddb4-4a68-8af2-27e0a316a1df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a026c3-prpbp" Oct 30 19:18:32.990681 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:32.990479 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4976184c-ddb4-4a68-8af2-27e0a316a1df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92a026c3-prpbp\" (UID: \"4976184c-ddb4-4a68-8af2-27e0a316a1df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a026c3-prpbp" Oct 30 19:18:33.091729 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.091655 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkkqg\" (UniqueName: \"kubernetes.io/projected/4976184c-ddb4-4a68-8af2-27e0a316a1df-kube-api-access-tkkqg\") pod \"maas-api-revoke-sub-92a026c3-prpbp\" (UID: \"4976184c-ddb4-4a68-8af2-27e0a316a1df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a026c3-prpbp" Oct 30 19:18:33.091729 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.091686 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4976184c-ddb4-4a68-8af2-27e0a316a1df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92a026c3-prpbp\" (UID: \"4976184c-ddb4-4a68-8af2-27e0a316a1df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a026c3-prpbp" Oct 30 19:18:33.091729 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.091721 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4976184c-ddb4-4a68-8af2-27e0a316a1df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92a026c3-prpbp\" (UID: \"4976184c-ddb4-4a68-8af2-27e0a316a1df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a026c3-prpbp" Oct 30 19:18:33.092411 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.092383 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4976184c-ddb4-4a68-8af2-27e0a316a1df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92a026c3-prpbp\" (UID: \"4976184c-ddb4-4a68-8af2-27e0a316a1df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a026c3-prpbp" Oct 30 19:18:33.094119 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.094097 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4976184c-ddb4-4a68-8af2-27e0a316a1df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92a026c3-prpbp\" (UID: \"4976184c-ddb4-4a68-8af2-27e0a316a1df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a026c3-prpbp" Oct 30 19:18:33.153060 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.153026 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkkqg\" (UniqueName: \"kubernetes.io/projected/4976184c-ddb4-4a68-8af2-27e0a316a1df-kube-api-access-tkkqg\") pod \"maas-api-revoke-sub-92a026c3-prpbp\" (UID: \"4976184c-ddb4-4a68-8af2-27e0a316a1df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a026c3-prpbp" Oct 30 19:18:33.374827 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.374741 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-226e0e2b-p2zxs"] Oct 30 19:18:33.378652 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.378630 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e0e2b-p2zxs" Oct 30 19:18:33.414655 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.414632 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a026c3-prpbp" Oct 30 19:18:33.443610 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.443583 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-226e0e2b-p2zxs"] Oct 30 19:18:33.498734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.497087 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnxj9\" (UniqueName: \"kubernetes.io/projected/8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5-kube-api-access-fnxj9\") pod \"maas-api-revoke-sub-226e0e2b-p2zxs\" (UID: \"8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e0e2b-p2zxs" Oct 30 19:18:33.498734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.497174 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-226e0e2b-p2zxs\" (UID: \"8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e0e2b-p2zxs" Oct 30 19:18:33.498734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.497267 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-226e0e2b-p2zxs\" (UID: \"8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e0e2b-p2zxs" Oct 30 19:18:33.561159 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.561093 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa70584f-fkhfl"] Oct 30 19:18:33.565026 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.565005 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa70584f-fkhfl" Oct 30 19:18:33.598092 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.598065 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-226e0e2b-p2zxs\" (UID: \"8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e0e2b-p2zxs" Oct 30 19:18:33.598265 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.598178 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnxj9\" (UniqueName: \"kubernetes.io/projected/8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5-kube-api-access-fnxj9\") pod \"maas-api-revoke-sub-226e0e2b-p2zxs\" (UID: \"8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e0e2b-p2zxs" Oct 30 19:18:33.598265 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.598208 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-226e0e2b-p2zxs\" (UID: \"8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e0e2b-p2zxs" Oct 30 19:18:33.598782 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.598764 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-226e0e2b-p2zxs\" (UID: \"8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e0e2b-p2zxs" Oct 30 19:18:33.600609 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.600583 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-226e0e2b-p2zxs\" (UID: \"8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e0e2b-p2zxs" Oct 30 19:18:33.623766 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.623732 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa70584f-fkhfl"] Oct 30 19:18:33.659053 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.658978 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnxj9\" (UniqueName: \"kubernetes.io/projected/8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5-kube-api-access-fnxj9\") pod \"maas-api-revoke-sub-226e0e2b-p2zxs\" (UID: \"8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e0e2b-p2zxs" Oct 30 19:18:33.691891 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.691865 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e0e2b-p2zxs" Oct 30 19:18:33.699533 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.699509 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfld9\" (UniqueName: \"kubernetes.io/projected/a97b9dea-b1ce-4818-931a-7060a9562bae-kube-api-access-rfld9\") pod \"maas-api-revoke-sub-aa70584f-fkhfl\" (UID: \"a97b9dea-b1ce-4818-931a-7060a9562bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa70584f-fkhfl" Oct 30 19:18:33.699617 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.699556 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97b9dea-b1ce-4818-931a-7060a9562bae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa70584f-fkhfl\" (UID: \"a97b9dea-b1ce-4818-931a-7060a9562bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa70584f-fkhfl" Oct 30 19:18:33.699749 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.699730 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97b9dea-b1ce-4818-931a-7060a9562bae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa70584f-fkhfl\" (UID: \"a97b9dea-b1ce-4818-931a-7060a9562bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa70584f-fkhfl" Oct 30 19:18:33.800635 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.800601 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfld9\" (UniqueName: \"kubernetes.io/projected/a97b9dea-b1ce-4818-931a-7060a9562bae-kube-api-access-rfld9\") pod \"maas-api-revoke-sub-aa70584f-fkhfl\" (UID: \"a97b9dea-b1ce-4818-931a-7060a9562bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa70584f-fkhfl" Oct 30 19:18:33.801097 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.800660 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97b9dea-b1ce-4818-931a-7060a9562bae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa70584f-fkhfl\" (UID: \"a97b9dea-b1ce-4818-931a-7060a9562bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa70584f-fkhfl" Oct 30 19:18:33.801097 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.800813 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97b9dea-b1ce-4818-931a-7060a9562bae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa70584f-fkhfl\" (UID: \"a97b9dea-b1ce-4818-931a-7060a9562bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa70584f-fkhfl" Oct 30 19:18:33.801525 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.801497 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97b9dea-b1ce-4818-931a-7060a9562bae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa70584f-fkhfl\" (UID: \"a97b9dea-b1ce-4818-931a-7060a9562bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa70584f-fkhfl" Oct 30 19:18:33.803428 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.803406 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97b9dea-b1ce-4818-931a-7060a9562bae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa70584f-fkhfl\" (UID: \"a97b9dea-b1ce-4818-931a-7060a9562bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa70584f-fkhfl" Oct 30 19:18:33.805797 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.805777 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92a026c3-prpbp"] Oct 30 19:18:33.808384 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:18:33.808357 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4976184c_ddb4_4a68_8af2_27e0a316a1df.slice/crio-f3825c85793d972ae17c5543aa8422580358c1840b820058fc89333536f6b22c WatchSource:0}: Error finding container f3825c85793d972ae17c5543aa8422580358c1840b820058fc89333536f6b22c: Status 404 returned error can't find the container with id f3825c85793d972ae17c5543aa8422580358c1840b820058fc89333536f6b22c Oct 30 19:18:33.862056 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.862028 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfld9\" (UniqueName: \"kubernetes.io/projected/a97b9dea-b1ce-4818-931a-7060a9562bae-kube-api-access-rfld9\") pod \"maas-api-revoke-sub-aa70584f-fkhfl\" (UID: \"a97b9dea-b1ce-4818-931a-7060a9562bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa70584f-fkhfl" Oct 30 19:18:33.878390 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.878366 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa70584f-fkhfl" Oct 30 19:18:33.937746 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:33.937266 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a026c3-prpbp" event={"ID":"4976184c-ddb4-4a68-8af2-27e0a316a1df","Type":"ContainerStarted","Data":"f3825c85793d972ae17c5543aa8422580358c1840b820058fc89333536f6b22c"} Oct 30 19:18:34.075258 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:34.075235 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-226e0e2b-p2zxs"] Oct 30 19:18:34.075835 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:18:34.075812 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0ce9b3_caab_4d5d_8b04_a5b3ccc89fb5.slice/crio-f1485abb8f00bf938d627cf0b7a39bfdde83a85c9de9adb976c1beab341fadec WatchSource:0}: Error finding container f1485abb8f00bf938d627cf0b7a39bfdde83a85c9de9adb976c1beab341fadec: Status 404 returned error can't find the container with id f1485abb8f00bf938d627cf0b7a39bfdde83a85c9de9adb976c1beab341fadec Oct 30 19:18:34.261000 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:34.260976 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa70584f-fkhfl"] Oct 30 19:18:34.319238 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:18:34.319197 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97b9dea_b1ce_4818_931a_7060a9562bae.slice/crio-810cc0c651bbc1a88b01d1000c3032cc026f3e2e85f26d95e1c1705830ff7e38 WatchSource:0}: Error finding container 810cc0c651bbc1a88b01d1000c3032cc026f3e2e85f26d95e1c1705830ff7e38: Status 404 returned error can't find the container with id 810cc0c651bbc1a88b01d1000c3032cc026f3e2e85f26d95e1c1705830ff7e38 Oct 30 19:18:34.945143 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:34.945099 4531 generic.go:356] "Generic (PLEG): container finished" podID="8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5" containerID="351e7f20a283827505a0194c1ca92224283e43a343da5982255be5914333a104" exitCode=0 Oct 30 19:18:34.945485 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:34.945166 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e0e2b-p2zxs" event={"ID":"8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5","Type":"ContainerDied","Data":"351e7f20a283827505a0194c1ca92224283e43a343da5982255be5914333a104"} Oct 30 19:18:34.945485 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:34.945222 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e0e2b-p2zxs" event={"ID":"8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5","Type":"ContainerStarted","Data":"f1485abb8f00bf938d627cf0b7a39bfdde83a85c9de9adb976c1beab341fadec"} Oct 30 19:18:34.946882 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:34.946864 4531 generic.go:356] "Generic (PLEG): container finished" podID="4976184c-ddb4-4a68-8af2-27e0a316a1df" containerID="f7b1fefe460c70baa949b039fd1eb732f9c58ba912b4a7fd1034ff41c3d4d4f9" exitCode=0 Oct 30 19:18:34.947000 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:34.946945 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a026c3-prpbp" event={"ID":"4976184c-ddb4-4a68-8af2-27e0a316a1df","Type":"ContainerDied","Data":"f7b1fefe460c70baa949b039fd1eb732f9c58ba912b4a7fd1034ff41c3d4d4f9"} Oct 30 19:18:34.948496 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:34.948472 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa70584f-fkhfl" event={"ID":"a97b9dea-b1ce-4818-931a-7060a9562bae","Type":"ContainerStarted","Data":"18396c05cf209c16aebac1eb57311ce60aee44d0e3563345407af645e5244533"} Oct 30 19:18:34.948603 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:34.948516 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa70584f-fkhfl" event={"ID":"a97b9dea-b1ce-4818-931a-7060a9562bae","Type":"ContainerStarted","Data":"810cc0c651bbc1a88b01d1000c3032cc026f3e2e85f26d95e1c1705830ff7e38"} Oct 30 19:18:35.183708 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:35.183592 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa70584f-fkhfl" podStartSLOduration=2.183575108 podStartE2EDuration="2.183575108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:35.182208333 +0000 UTC m=+2675.209313293" watchObservedRunningTime="2026-10-30 19:18:35.183575108 +0000 UTC m=+2675.210680068" Oct 30 19:18:35.956181 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:35.956112 4531 generic.go:356] "Generic (PLEG): container finished" podID="a97b9dea-b1ce-4818-931a-7060a9562bae" containerID="18396c05cf209c16aebac1eb57311ce60aee44d0e3563345407af645e5244533" exitCode=0 Oct 30 19:18:35.956181 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:35.956174 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa70584f-fkhfl" event={"ID":"a97b9dea-b1ce-4818-931a-7060a9562bae","Type":"ContainerDied","Data":"18396c05cf209c16aebac1eb57311ce60aee44d0e3563345407af645e5244533"} Oct 30 19:18:36.109172 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.105656 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e0e2b-p2zxs" Oct 30 19:18:36.126314 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.126284 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5-maas-api-xks-ca\") pod \"8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5\" (UID: \"8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5\") " Oct 30 19:18:36.126475 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.126446 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5-maas-api-openshift-ca\") pod \"8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5\" (UID: \"8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5\") " Oct 30 19:18:36.126547 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.126496 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5-kube-api-access-fnxj9\" (UniqueName: \"kubernetes.io/projected/8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5-kube-api-access-fnxj9\") pod \"8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5\" (UID: \"8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5\") " Oct 30 19:18:36.126860 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.126826 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5-maas-api-openshift-ca" pod "8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5" (UID: "8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:18:36.126989 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.126957 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:36.128487 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.128460 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5-maas-api-xks-ca" pod "8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5" (UID: "8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:18:36.128795 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.128774 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5-kube-api-access-fnxj9" pod "8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5" (UID: "8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5"). InnerVolumeSpecName "kube-api-access-fnxj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:18:36.137982 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.137961 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a026c3-prpbp" Oct 30 19:18:36.228285 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.228252 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4976184c-ddb4-4a68-8af2-27e0a316a1df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4976184c-ddb4-4a68-8af2-27e0a316a1df-maas-api-openshift-ca\") pod \"4976184c-ddb4-4a68-8af2-27e0a316a1df\" (UID: \"4976184c-ddb4-4a68-8af2-27e0a316a1df\") " Oct 30 19:18:36.228429 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.228412 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4976184c-ddb4-4a68-8af2-27e0a316a1df-kube-api-access-tkkqg\" (UniqueName: \"kubernetes.io/projected/4976184c-ddb4-4a68-8af2-27e0a316a1df-kube-api-access-tkkqg\") pod \"4976184c-ddb4-4a68-8af2-27e0a316a1df\" (UID: \"4976184c-ddb4-4a68-8af2-27e0a316a1df\") " Oct 30 19:18:36.228496 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.228464 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4976184c-ddb4-4a68-8af2-27e0a316a1df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4976184c-ddb4-4a68-8af2-27e0a316a1df-maas-api-xks-ca\") pod \"4976184c-ddb4-4a68-8af2-27e0a316a1df\" (UID: \"4976184c-ddb4-4a68-8af2-27e0a316a1df\") " Oct 30 19:18:36.228957 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.228878 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnxj9\" (UniqueName: \"kubernetes.io/projected/8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5-kube-api-access-fnxj9\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:36.228957 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.228903 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:36.229156 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.228999 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4976184c-ddb4-4a68-8af2-27e0a316a1df-maas-api-openshift-ca" pod "4976184c-ddb4-4a68-8af2-27e0a316a1df" (UID: "4976184c-ddb4-4a68-8af2-27e0a316a1df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:18:36.231451 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.231425 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4976184c-ddb4-4a68-8af2-27e0a316a1df-maas-api-xks-ca" pod "4976184c-ddb4-4a68-8af2-27e0a316a1df" (UID: "4976184c-ddb4-4a68-8af2-27e0a316a1df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:18:36.231451 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.231441 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4976184c-ddb4-4a68-8af2-27e0a316a1df-kube-api-access-tkkqg" pod "4976184c-ddb4-4a68-8af2-27e0a316a1df" (UID: "4976184c-ddb4-4a68-8af2-27e0a316a1df"). InnerVolumeSpecName "kube-api-access-tkkqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:18:36.330481 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.330452 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4976184c-ddb4-4a68-8af2-27e0a316a1df-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:36.330481 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.330477 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkkqg\" (UniqueName: \"kubernetes.io/projected/4976184c-ddb4-4a68-8af2-27e0a316a1df-kube-api-access-tkkqg\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:36.330481 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.330487 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4976184c-ddb4-4a68-8af2-27e0a316a1df-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:36.963588 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.963548 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e0e2b-p2zxs" event={"ID":"8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5","Type":"ContainerDied","Data":"f1485abb8f00bf938d627cf0b7a39bfdde83a85c9de9adb976c1beab341fadec"} Oct 30 19:18:36.964050 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.963614 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1485abb8f00bf938d627cf0b7a39bfdde83a85c9de9adb976c1beab341fadec" Oct 30 19:18:36.964050 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.963560 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-226e0e2b-p2zxs" Oct 30 19:18:36.971051 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.968377 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a026c3-prpbp" event={"ID":"4976184c-ddb4-4a68-8af2-27e0a316a1df","Type":"ContainerDied","Data":"f3825c85793d972ae17c5543aa8422580358c1840b820058fc89333536f6b22c"} Oct 30 19:18:36.971051 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.968479 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3825c85793d972ae17c5543aa8422580358c1840b820058fc89333536f6b22c" Oct 30 19:18:36.971051 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.969279 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a026c3-prpbp" Oct 30 19:18:36.991011 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.990973 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9"] Oct 30 19:18:36.991304 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:36.991259 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9" podUID="667fc3fe-bd84-4db3-9ab4-cca263a6d1cb" containerName="payload-processing" containerID="cri-o://91fbd3109eeaf0d90bd4ca362379445aee7a5989375cffa3074050dcb8e6fc5a" gracePeriod=30 Oct 30 19:18:37.047013 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.046976 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c"] Oct 30 19:18:37.047340 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.047288 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c" podUID="18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3" containerName="payload-pre-processing" containerID="cri-o://e27f2f718e6d0f1a64f1706e585d8aea6e12468c2954ba9eda2c2526302ec8bf" gracePeriod=30 Oct 30 19:18:37.179215 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.179185 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa70584f-fkhfl" Oct 30 19:18:37.239709 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.239685 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a97b9dea-b1ce-4818-931a-7060a9562bae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97b9dea-b1ce-4818-931a-7060a9562bae-maas-api-xks-ca\") pod \"a97b9dea-b1ce-4818-931a-7060a9562bae\" (UID: \"a97b9dea-b1ce-4818-931a-7060a9562bae\") " Oct 30 19:18:37.239831 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.239740 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a97b9dea-b1ce-4818-931a-7060a9562bae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97b9dea-b1ce-4818-931a-7060a9562bae-maas-api-openshift-ca\") pod \"a97b9dea-b1ce-4818-931a-7060a9562bae\" (UID: \"a97b9dea-b1ce-4818-931a-7060a9562bae\") " Oct 30 19:18:37.239898 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.239885 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a97b9dea-b1ce-4818-931a-7060a9562bae-kube-api-access-rfld9\" (UniqueName: \"kubernetes.io/projected/a97b9dea-b1ce-4818-931a-7060a9562bae-kube-api-access-rfld9\") pod \"a97b9dea-b1ce-4818-931a-7060a9562bae\" (UID: \"a97b9dea-b1ce-4818-931a-7060a9562bae\") " Oct 30 19:18:37.240207 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.240168 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a97b9dea-b1ce-4818-931a-7060a9562bae-maas-api-openshift-ca" pod "a97b9dea-b1ce-4818-931a-7060a9562bae" (UID: "a97b9dea-b1ce-4818-931a-7060a9562bae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:18:37.242171 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.242147 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a97b9dea-b1ce-4818-931a-7060a9562bae-maas-api-xks-ca" pod "a97b9dea-b1ce-4818-931a-7060a9562bae" (UID: "a97b9dea-b1ce-4818-931a-7060a9562bae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:18:37.242396 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.242372 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a97b9dea-b1ce-4818-931a-7060a9562bae-kube-api-access-rfld9" pod "a97b9dea-b1ce-4818-931a-7060a9562bae" (UID: "a97b9dea-b1ce-4818-931a-7060a9562bae"). InnerVolumeSpecName "kube-api-access-rfld9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:18:37.330580 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.330557 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9" Oct 30 19:18:37.334123 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.334106 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c" Oct 30 19:18:37.340594 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.340576 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/667fc3fe-bd84-4db3-9ab4-cca263a6d1cb-kube-api-access-z8n42\" (UniqueName: \"kubernetes.io/projected/667fc3fe-bd84-4db3-9ab4-cca263a6d1cb-kube-api-access-z8n42\") pod \"667fc3fe-bd84-4db3-9ab4-cca263a6d1cb\" (UID: \"667fc3fe-bd84-4db3-9ab4-cca263a6d1cb\") " Oct 30 19:18:37.340680 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.340607 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3-kube-api-access-btqv4\" (UniqueName: \"kubernetes.io/projected/18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3-kube-api-access-btqv4\") pod \"18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3\" (UID: \"18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3\") " Oct 30 19:18:37.340680 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.340626 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3-plugins-config-volume\") pod \"18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3\" (UID: \"18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3\") " Oct 30 19:18:37.340802 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.340696 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/667fc3fe-bd84-4db3-9ab4-cca263a6d1cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/667fc3fe-bd84-4db3-9ab4-cca263a6d1cb-plugins-config-volume\") pod \"667fc3fe-bd84-4db3-9ab4-cca263a6d1cb\" (UID: \"667fc3fe-bd84-4db3-9ab4-cca263a6d1cb\") " Oct 30 19:18:37.341041 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.341017 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3-plugins-config-volume" pod "18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3" (UID: "18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:18:37.341152 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.341102 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/667fc3fe-bd84-4db3-9ab4-cca263a6d1cb-plugins-config-volume" pod "667fc3fe-bd84-4db3-9ab4-cca263a6d1cb" (UID: "667fc3fe-bd84-4db3-9ab4-cca263a6d1cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:18:37.341290 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.341246 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfld9\" (UniqueName: \"kubernetes.io/projected/a97b9dea-b1ce-4818-931a-7060a9562bae-kube-api-access-rfld9\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:37.341290 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.341263 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:37.341290 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.341272 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/667fc3fe-bd84-4db3-9ab4-cca263a6d1cb-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:37.341290 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.341281 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97b9dea-b1ce-4818-931a-7060a9562bae-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:37.341290 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.341289 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97b9dea-b1ce-4818-931a-7060a9562bae-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:37.342865 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.342842 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667fc3fe-bd84-4db3-9ab4-cca263a6d1cb-kube-api-access-z8n42" pod "667fc3fe-bd84-4db3-9ab4-cca263a6d1cb" (UID: "667fc3fe-bd84-4db3-9ab4-cca263a6d1cb"). InnerVolumeSpecName "kube-api-access-z8n42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:18:37.342957 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.342876 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3-kube-api-access-btqv4" pod "18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3" (UID: "18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3"). InnerVolumeSpecName "kube-api-access-btqv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:18:37.442439 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.442409 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8n42\" (UniqueName: \"kubernetes.io/projected/667fc3fe-bd84-4db3-9ab4-cca263a6d1cb-kube-api-access-z8n42\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:37.442560 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.442443 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btqv4\" (UniqueName: \"kubernetes.io/projected/18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3-kube-api-access-btqv4\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:37.975735 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.975698 4531 generic.go:356] "Generic (PLEG): container finished" podID="667fc3fe-bd84-4db3-9ab4-cca263a6d1cb" containerID="91fbd3109eeaf0d90bd4ca362379445aee7a5989375cffa3074050dcb8e6fc5a" exitCode=0 Oct 30 19:18:37.976254 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.975787 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9" event={"ID":"667fc3fe-bd84-4db3-9ab4-cca263a6d1cb","Type":"ContainerDied","Data":"91fbd3109eeaf0d90bd4ca362379445aee7a5989375cffa3074050dcb8e6fc5a"} Oct 30 19:18:37.976254 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.975797 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9" Oct 30 19:18:37.976254 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.975837 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9" event={"ID":"667fc3fe-bd84-4db3-9ab4-cca263a6d1cb","Type":"ContainerDied","Data":"3882dad77f85ce0e510cdd160b3807b95b1b7d566a6c8f14908e73e1b5f14403"} Oct 30 19:18:37.976254 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.975858 4531 scope.go:122] "RemoveContainer" containerID="91fbd3109eeaf0d90bd4ca362379445aee7a5989375cffa3074050dcb8e6fc5a" Oct 30 19:18:37.977669 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.977650 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa70584f-fkhfl" Oct 30 19:18:37.977945 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.977910 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa70584f-fkhfl" event={"ID":"a97b9dea-b1ce-4818-931a-7060a9562bae","Type":"ContainerDied","Data":"810cc0c651bbc1a88b01d1000c3032cc026f3e2e85f26d95e1c1705830ff7e38"} Oct 30 19:18:37.978075 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.977956 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="810cc0c651bbc1a88b01d1000c3032cc026f3e2e85f26d95e1c1705830ff7e38" Oct 30 19:18:37.979252 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.979230 4531 generic.go:356] "Generic (PLEG): container finished" podID="18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3" containerID="e27f2f718e6d0f1a64f1706e585d8aea6e12468c2954ba9eda2c2526302ec8bf" exitCode=0 Oct 30 19:18:37.979354 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.979339 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c" event={"ID":"18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3","Type":"ContainerDied","Data":"e27f2f718e6d0f1a64f1706e585d8aea6e12468c2954ba9eda2c2526302ec8bf"} Oct 30 19:18:37.979399 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.979382 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c" event={"ID":"18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3","Type":"ContainerDied","Data":"59a47501acfc44dd2192f041bd129b8f1073828f29fc3e23e6fedbe929407b8f"} Oct 30 19:18:37.979463 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.979451 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c" Oct 30 19:18:37.992682 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.992667 4531 scope.go:122] "RemoveContainer" containerID="91fbd3109eeaf0d90bd4ca362379445aee7a5989375cffa3074050dcb8e6fc5a" Oct 30 19:18:37.992998 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:18:37.992974 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91fbd3109eeaf0d90bd4ca362379445aee7a5989375cffa3074050dcb8e6fc5a\": container with ID starting with 91fbd3109eeaf0d90bd4ca362379445aee7a5989375cffa3074050dcb8e6fc5a not found: ID does not exist" containerID="91fbd3109eeaf0d90bd4ca362379445aee7a5989375cffa3074050dcb8e6fc5a" Oct 30 19:18:37.993077 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.993006 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91fbd3109eeaf0d90bd4ca362379445aee7a5989375cffa3074050dcb8e6fc5a"} err="failed to get container status \"91fbd3109eeaf0d90bd4ca362379445aee7a5989375cffa3074050dcb8e6fc5a\": rpc error: code = NotFound desc = could not find container \"91fbd3109eeaf0d90bd4ca362379445aee7a5989375cffa3074050dcb8e6fc5a\": container with ID starting with 91fbd3109eeaf0d90bd4ca362379445aee7a5989375cffa3074050dcb8e6fc5a not found: ID does not exist" Oct 30 19:18:37.993077 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:37.993027 4531 scope.go:122] "RemoveContainer" containerID="e27f2f718e6d0f1a64f1706e585d8aea6e12468c2954ba9eda2c2526302ec8bf" Oct 30 19:18:38.002436 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:38.002416 4531 scope.go:122] "RemoveContainer" containerID="e27f2f718e6d0f1a64f1706e585d8aea6e12468c2954ba9eda2c2526302ec8bf" Oct 30 19:18:38.002675 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:18:38.002658 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e27f2f718e6d0f1a64f1706e585d8aea6e12468c2954ba9eda2c2526302ec8bf\": container with ID starting with e27f2f718e6d0f1a64f1706e585d8aea6e12468c2954ba9eda2c2526302ec8bf not found: ID does not exist" containerID="e27f2f718e6d0f1a64f1706e585d8aea6e12468c2954ba9eda2c2526302ec8bf" Oct 30 19:18:38.002743 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:38.002680 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e27f2f718e6d0f1a64f1706e585d8aea6e12468c2954ba9eda2c2526302ec8bf"} err="failed to get container status \"e27f2f718e6d0f1a64f1706e585d8aea6e12468c2954ba9eda2c2526302ec8bf\": rpc error: code = NotFound desc = could not find container \"e27f2f718e6d0f1a64f1706e585d8aea6e12468c2954ba9eda2c2526302ec8bf\": container with ID starting with e27f2f718e6d0f1a64f1706e585d8aea6e12468c2954ba9eda2c2526302ec8bf not found: ID does not exist" Oct 30 19:18:38.141157 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:38.141115 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9"] Oct 30 19:18:38.145141 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:38.145102 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-91cb32-794f76cc68-vvtb9"] Oct 30 19:18:38.253306 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:38.253261 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c"] Oct 30 19:18:38.257453 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:38.257430 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-91cb32-6679c67d8c-mtc6c"] Oct 30 19:18:38.711600 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:38.711502 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw"] Oct 30 19:18:38.714546 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:38.714523 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-91cb32-6rwkw"] Oct 30 19:18:40.099260 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:40.099230 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3" path="/var/lib/kubelet/pods/18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3/volumes" Oct 30 19:18:40.099638 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:40.099590 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="667fc3fe-bd84-4db3-9ab4-cca263a6d1cb" path="/var/lib/kubelet/pods/667fc3fe-bd84-4db3-9ab4-cca263a6d1cb/volumes" Oct 30 19:18:40.099942 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:40.099928 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de0f819c-c2c3-4022-afee-425bd1a73c69" path="/var/lib/kubelet/pods/de0f819c-c2c3-4022-afee-425bd1a73c69/volumes" Oct 30 19:18:44.163170 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.163140 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3500cb0-jbxdh"] Oct 30 19:18:44.163663 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.163521 4531 cpu_manager.go:414] "containerMap: removing container" podUID="667fc3fe-bd84-4db3-9ab4-cca263a6d1cb" containerName="payload-processing" Oct 30 19:18:44.163663 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.163533 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="667fc3fe-bd84-4db3-9ab4-cca263a6d1cb" containerName="payload-processing" Oct 30 19:18:44.163663 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.163552 4531 cpu_manager.go:414] "containerMap: removing container" podUID="18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3" containerName="payload-pre-processing" Oct 30 19:18:44.163663 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.163558 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3" containerName="payload-pre-processing" Oct 30 19:18:44.163663 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.163567 4531 cpu_manager.go:414] "containerMap: removing container" podUID="4976184c-ddb4-4a68-8af2-27e0a316a1df" containerName="revoke-keys" Oct 30 19:18:44.163663 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.163572 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4976184c-ddb4-4a68-8af2-27e0a316a1df" containerName="revoke-keys" Oct 30 19:18:44.163663 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.163597 4531 cpu_manager.go:414] "containerMap: removing container" podUID="8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5" containerName="revoke-keys" Oct 30 19:18:44.163663 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.163603 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5" containerName="revoke-keys" Oct 30 19:18:44.163663 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.163608 4531 cpu_manager.go:414] "containerMap: removing container" podUID="a97b9dea-b1ce-4818-931a-7060a9562bae" containerName="revoke-keys" Oct 30 19:18:44.163663 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.163612 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a97b9dea-b1ce-4818-931a-7060a9562bae" containerName="revoke-keys" Oct 30 19:18:44.164007 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.163674 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="667fc3fe-bd84-4db3-9ab4-cca263a6d1cb" containerName="payload-processing" Oct 30 19:18:44.164007 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.163684 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="18bc7546-55ad-4f84-9eb6-4e2b3cf1c4f3" containerName="payload-pre-processing" Oct 30 19:18:44.164007 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.163691 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4976184c-ddb4-4a68-8af2-27e0a316a1df" containerName="revoke-keys" Oct 30 19:18:44.164007 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.163702 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5" containerName="revoke-keys" Oct 30 19:18:44.164007 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.163708 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a97b9dea-b1ce-4818-931a-7060a9562bae" containerName="revoke-keys" Oct 30 19:18:44.168583 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.168563 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3500cb0-jbxdh" Oct 30 19:18:44.214013 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.213986 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c10d4e49-7a07-49ea-9707-66956ddd6ceb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3500cb0-jbxdh\" (UID: \"c10d4e49-7a07-49ea-9707-66956ddd6ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3500cb0-jbxdh" Oct 30 19:18:44.214157 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.214037 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c10d4e49-7a07-49ea-9707-66956ddd6ceb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3500cb0-jbxdh\" (UID: \"c10d4e49-7a07-49ea-9707-66956ddd6ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3500cb0-jbxdh" Oct 30 19:18:44.214223 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.214202 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2fks\" (UniqueName: \"kubernetes.io/projected/c10d4e49-7a07-49ea-9707-66956ddd6ceb-kube-api-access-q2fks\") pod \"maas-api-revoke-sub-c3500cb0-jbxdh\" (UID: \"c10d4e49-7a07-49ea-9707-66956ddd6ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3500cb0-jbxdh" Oct 30 19:18:44.221888 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.221851 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:18:44.227054 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.226531 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3500cb0-jbxdh"] Oct 30 19:18:44.271228 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.271197 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:18:44.271228 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.271220 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:18:44.271429 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.271278 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:18:44.314819 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.314793 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2fks\" (UniqueName: \"kubernetes.io/projected/c10d4e49-7a07-49ea-9707-66956ddd6ceb-kube-api-access-q2fks\") pod \"maas-api-revoke-sub-c3500cb0-jbxdh\" (UID: \"c10d4e49-7a07-49ea-9707-66956ddd6ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3500cb0-jbxdh" Oct 30 19:18:44.314951 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.314862 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c10d4e49-7a07-49ea-9707-66956ddd6ceb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3500cb0-jbxdh\" (UID: \"c10d4e49-7a07-49ea-9707-66956ddd6ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3500cb0-jbxdh" Oct 30 19:18:44.314951 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.314903 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c10d4e49-7a07-49ea-9707-66956ddd6ceb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3500cb0-jbxdh\" (UID: \"c10d4e49-7a07-49ea-9707-66956ddd6ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3500cb0-jbxdh" Oct 30 19:18:44.315498 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.315474 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c10d4e49-7a07-49ea-9707-66956ddd6ceb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3500cb0-jbxdh\" (UID: \"c10d4e49-7a07-49ea-9707-66956ddd6ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3500cb0-jbxdh" Oct 30 19:18:44.317424 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.317398 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c10d4e49-7a07-49ea-9707-66956ddd6ceb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3500cb0-jbxdh\" (UID: \"c10d4e49-7a07-49ea-9707-66956ddd6ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3500cb0-jbxdh" Oct 30 19:18:44.376174 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.376149 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2fks\" (UniqueName: \"kubernetes.io/projected/c10d4e49-7a07-49ea-9707-66956ddd6ceb-kube-api-access-q2fks\") pod \"maas-api-revoke-sub-c3500cb0-jbxdh\" (UID: \"c10d4e49-7a07-49ea-9707-66956ddd6ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3500cb0-jbxdh" Oct 30 19:18:44.481678 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.481614 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3500cb0-jbxdh" Oct 30 19:18:44.867031 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:44.865764 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3500cb0-jbxdh"] Oct 30 19:18:45.032906 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:45.032876 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3500cb0-jbxdh" event={"ID":"c10d4e49-7a07-49ea-9707-66956ddd6ceb","Type":"ContainerStarted","Data":"6c4294d9be044d30356b131e075c7f256793e97ad09a95c3ddd0bdf2466ae069"} Oct 30 19:18:46.046313 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:46.046275 4531 generic.go:356] "Generic (PLEG): container finished" podID="c10d4e49-7a07-49ea-9707-66956ddd6ceb" containerID="468dd05d379d653b3c6d8110e5a070d6358dfef7e351cb46575c6c963c6dfcdc" exitCode=0 Oct 30 19:18:46.046810 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:46.046389 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3500cb0-jbxdh" event={"ID":"c10d4e49-7a07-49ea-9707-66956ddd6ceb","Type":"ContainerDied","Data":"468dd05d379d653b3c6d8110e5a070d6358dfef7e351cb46575c6c963c6dfcdc"} Oct 30 19:18:47.205374 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:47.205347 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3500cb0-jbxdh" Oct 30 19:18:47.247468 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:47.247435 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c10d4e49-7a07-49ea-9707-66956ddd6ceb-kube-api-access-q2fks\" (UniqueName: \"kubernetes.io/projected/c10d4e49-7a07-49ea-9707-66956ddd6ceb-kube-api-access-q2fks\") pod \"c10d4e49-7a07-49ea-9707-66956ddd6ceb\" (UID: \"c10d4e49-7a07-49ea-9707-66956ddd6ceb\") " Oct 30 19:18:47.247653 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:47.247524 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c10d4e49-7a07-49ea-9707-66956ddd6ceb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c10d4e49-7a07-49ea-9707-66956ddd6ceb-maas-api-openshift-ca\") pod \"c10d4e49-7a07-49ea-9707-66956ddd6ceb\" (UID: \"c10d4e49-7a07-49ea-9707-66956ddd6ceb\") " Oct 30 19:18:47.247732 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:47.247668 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c10d4e49-7a07-49ea-9707-66956ddd6ceb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c10d4e49-7a07-49ea-9707-66956ddd6ceb-maas-api-xks-ca\") pod \"c10d4e49-7a07-49ea-9707-66956ddd6ceb\" (UID: \"c10d4e49-7a07-49ea-9707-66956ddd6ceb\") " Oct 30 19:18:47.247898 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:47.247871 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c10d4e49-7a07-49ea-9707-66956ddd6ceb-maas-api-openshift-ca" pod "c10d4e49-7a07-49ea-9707-66956ddd6ceb" (UID: "c10d4e49-7a07-49ea-9707-66956ddd6ceb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:18:47.248093 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:47.248076 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c10d4e49-7a07-49ea-9707-66956ddd6ceb-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:47.249899 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:47.249865 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c10d4e49-7a07-49ea-9707-66956ddd6ceb-maas-api-xks-ca" pod "c10d4e49-7a07-49ea-9707-66956ddd6ceb" (UID: "c10d4e49-7a07-49ea-9707-66956ddd6ceb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:18:47.249995 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:47.249947 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10d4e49-7a07-49ea-9707-66956ddd6ceb-kube-api-access-q2fks" pod "c10d4e49-7a07-49ea-9707-66956ddd6ceb" (UID: "c10d4e49-7a07-49ea-9707-66956ddd6ceb"). InnerVolumeSpecName "kube-api-access-q2fks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:18:47.349715 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:47.349629 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c10d4e49-7a07-49ea-9707-66956ddd6ceb-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:47.349715 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:47.349668 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2fks\" (UniqueName: \"kubernetes.io/projected/c10d4e49-7a07-49ea-9707-66956ddd6ceb-kube-api-access-q2fks\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:18:48.060465 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:48.060431 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3500cb0-jbxdh" Oct 30 19:18:48.060668 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:48.060473 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3500cb0-jbxdh" event={"ID":"c10d4e49-7a07-49ea-9707-66956ddd6ceb","Type":"ContainerDied","Data":"6c4294d9be044d30356b131e075c7f256793e97ad09a95c3ddd0bdf2466ae069"} Oct 30 19:18:48.060668 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:18:48.060528 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c4294d9be044d30356b131e075c7f256793e97ad09a95c3ddd0bdf2466ae069" Oct 30 19:18:56.397741 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:18:56.397704 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:19:02.072485 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:02.072456 4531 scope.go:122] "RemoveContainer" containerID="dcebb7b1348d6f7eaf2e71c2e341576b2186f15c07b2e29057820e210c2743f7" Oct 30 19:19:17.076165 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:17.076111 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb"] Oct 30 19:19:17.076637 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:17.076557 4531 cpu_manager.go:414] "containerMap: removing container" podUID="c10d4e49-7a07-49ea-9707-66956ddd6ceb" containerName="revoke-keys" Oct 30 19:19:17.076637 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:17.076568 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c10d4e49-7a07-49ea-9707-66956ddd6ceb" containerName="revoke-keys" Oct 30 19:19:17.076712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:17.076640 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c10d4e49-7a07-49ea-9707-66956ddd6ceb" containerName="revoke-keys" Oct 30 19:19:17.081197 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:17.081178 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb" Oct 30 19:19:17.182846 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:17.182815 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-19f6ea\"" Oct 30 19:19:17.182846 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:17.182819 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-19f6ea-dockercfg-5kbvf\"" Oct 30 19:19:17.185522 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:17.185491 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2znlx\" (UniqueName: \"kubernetes.io/projected/bc9144a3-39f6-4066-b677-0261f5f9fe3e-kube-api-access-2znlx\") pod \"payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb\" (UID: \"bc9144a3-39f6-4066-b677-0261f5f9fe3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb" Oct 30 19:19:17.185675 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:17.185603 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9144a3-39f6-4066-b677-0261f5f9fe3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb\" (UID: \"bc9144a3-39f6-4066-b677-0261f5f9fe3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb" Oct 30 19:19:17.186450 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:17.186428 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb"] Oct 30 19:19:17.286751 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:17.286711 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2znlx\" (UniqueName: \"kubernetes.io/projected/bc9144a3-39f6-4066-b677-0261f5f9fe3e-kube-api-access-2znlx\") pod \"payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb\" (UID: \"bc9144a3-39f6-4066-b677-0261f5f9fe3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb" Oct 30 19:19:17.286976 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:17.286876 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9144a3-39f6-4066-b677-0261f5f9fe3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb\" (UID: \"bc9144a3-39f6-4066-b677-0261f5f9fe3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb" Oct 30 19:19:17.287875 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:17.287848 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9144a3-39f6-4066-b677-0261f5f9fe3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb\" (UID: \"bc9144a3-39f6-4066-b677-0261f5f9fe3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb" Oct 30 19:19:17.348030 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:17.347953 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2znlx\" (UniqueName: \"kubernetes.io/projected/bc9144a3-39f6-4066-b677-0261f5f9fe3e-kube-api-access-2znlx\") pod \"payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb\" (UID: \"bc9144a3-39f6-4066-b677-0261f5f9fe3e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb" Oct 30 19:19:17.393508 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:17.393476 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb" Oct 30 19:19:17.826028 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:17.825996 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb"] Oct 30 19:19:17.831756 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:19:17.831721 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9144a3_39f6_4066_b677_0261f5f9fe3e.slice/crio-954a7e255d4c7293bf6f45c0edfa0c02a37a5e60257c0faf8ec18d84986cafe8 WatchSource:0}: Error finding container 954a7e255d4c7293bf6f45c0edfa0c02a37a5e60257c0faf8ec18d84986cafe8: Status 404 returned error can't find the container with id 954a7e255d4c7293bf6f45c0edfa0c02a37a5e60257c0faf8ec18d84986cafe8 Oct 30 19:19:18.255147 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:18.255094 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb" event={"ID":"bc9144a3-39f6-4066-b677-0261f5f9fe3e","Type":"ContainerStarted","Data":"d26ea88bdb144b2cea10d8d390815b9159114e6afe5586d69e2c3e33649e5727"} Oct 30 19:19:18.255546 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:18.255171 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb" event={"ID":"bc9144a3-39f6-4066-b677-0261f5f9fe3e","Type":"ContainerStarted","Data":"954a7e255d4c7293bf6f45c0edfa0c02a37a5e60257c0faf8ec18d84986cafe8"} Oct 30 19:19:18.255546 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:18.255200 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb" Oct 30 19:19:18.373041 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:18.372987 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb" podStartSLOduration=1.372973495 podStartE2EDuration="1.372973495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:18.370698873 +0000 UTC m=+2718.397803836" watchObservedRunningTime="2026-10-30 19:19:18.372973495 +0000 UTC m=+2718.400078455" Oct 30 19:19:26.396200 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:19:26.396163 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:19:29.264106 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:29.264080 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb" Oct 30 19:19:41.051952 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:41.051877 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-575cced8-2zvz6"] Oct 30 19:19:41.100564 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:41.100530 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-575cced8-2zvz6"] Oct 30 19:19:42.098373 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:42.098334 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ed85520-ff89-44e0-a03e-8aa423fe3cf5" path="/var/lib/kubelet/pods/9ed85520-ff89-44e0-a03e-8aa423fe3cf5/volumes" Oct 30 19:19:51.053831 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:51.053788 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6d3eb80-n9c2t"] Oct 30 19:19:51.101532 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:51.101503 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6d3eb80-n9c2t"] Oct 30 19:19:52.104998 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:52.104962 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84ef0d55-d199-4b86-8113-36bbbc61f614" path="/var/lib/kubelet/pods/84ef0d55-d199-4b86-8113-36bbbc61f614/volumes" Oct 30 19:19:56.397139 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:19:56.397086 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:19:59.060937 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:59.060897 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7064831-hpp87"] Oct 30 19:19:59.063974 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:19:59.063949 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7064831-hpp87"] Oct 30 19:20:00.097731 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:00.097701 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d486a241-c368-4e95-ad58-e06f5e41ef24" path="/var/lib/kubelet/pods/d486a241-c368-4e95-ad58-e06f5e41ef24/volumes" Oct 30 19:20:02.110805 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:02.110766 4531 scope.go:122] "RemoveContainer" containerID="fba1f919b6513d3568d5243bcd458e5ed22b6d65d8798d0f35a1c6f34fbf194f" Oct 30 19:20:02.121942 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:02.121917 4531 scope.go:122] "RemoveContainer" containerID="1f40e9b529530d7e7268a6576b4b7b2718a9e9e461e54b2b86496a9007bef38d" Oct 30 19:20:26.400625 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:20:26.400573 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:20:32.059198 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:32.059156 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87bb966f-sj9k2"] Oct 30 19:20:32.108371 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:32.108330 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87bb966f-sj9k2"] Oct 30 19:20:34.098970 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:34.098932 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54984f9c-9fea-474f-b01e-31054a052892" path="/var/lib/kubelet/pods/54984f9c-9fea-474f-b01e-31054a052892/volumes" Oct 30 19:20:39.135297 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.135239 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x"] Oct 30 19:20:39.140089 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.140057 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.192907 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.192867 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-dc5cdd-openshift-default-dockercfg-sw8kz\"" Oct 30 19:20:39.247277 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.247234 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x"] Oct 30 19:20:39.256615 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.256558 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88sz7\" (UniqueName: \"kubernetes.io/projected/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-kube-api-access-88sz7\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.256849 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.256631 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-token\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.256849 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.256735 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-podinfo\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.256986 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.256847 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-envoy\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.256986 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.256888 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-workload-socket\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.257110 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.256971 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-credential-socket\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.257110 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.257031 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-data\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.257283 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.257183 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-workload-certs\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.257283 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.257225 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istiod-ca-cert\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.358742 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.358694 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-workload-certs\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.358959 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.358750 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istiod-ca-cert\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.358959 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.358800 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88sz7\" (UniqueName: \"kubernetes.io/projected/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-kube-api-access-88sz7\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.359084 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.358955 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-token\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.359084 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.359034 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-podinfo\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.359272 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.359155 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-envoy\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.359272 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.359123 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-workload-certs\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.359272 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.359236 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-workload-socket\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.359272 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.359263 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-credential-socket\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.359500 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.359305 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-data\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.359589 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.359558 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-workload-socket\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.359651 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.359600 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-credential-socket\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.359651 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.359630 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istiod-ca-cert\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.359735 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.359665 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-data\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.361583 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.361560 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-envoy\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.361861 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.361841 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-podinfo\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.421063 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.420986 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-token\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.421250 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.421182 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88sz7\" (UniqueName: \"kubernetes.io/projected/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-kube-api-access-88sz7\") pod \"e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.456447 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.456408 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:39.897673 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.897641 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x"] Oct 30 19:20:39.899764 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:20:39.899732 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0484c6c5_9f9e_408c_9974_a4f5f85b6bea.slice/crio-c7c7087b07f87ccf59627c2b4ce7dc213d7e81cbb2790b0e6475fe887da00ede WatchSource:0}: Error finding container c7c7087b07f87ccf59627c2b4ce7dc213d7e81cbb2790b0e6475fe887da00ede: Status 404 returned error can't find the container with id c7c7087b07f87ccf59627c2b4ce7dc213d7e81cbb2790b0e6475fe887da00ede Oct 30 19:20:39.902364 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.902328 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:20:39.902462 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.902401 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:20:39.902462 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:39.902430 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:20:40.806852 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:40.806813 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" event={"ID":"0484c6c5-9f9e-408c-9974-a4f5f85b6bea","Type":"ContainerStarted","Data":"a41389ff7303cf57aa56f4ba914e97864d722a33b72ab90c2a940309552f90bb"} Oct 30 19:20:40.807369 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:40.806871 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" event={"ID":"0484c6c5-9f9e-408c-9974-a4f5f85b6bea","Type":"ContainerStarted","Data":"c7c7087b07f87ccf59627c2b4ce7dc213d7e81cbb2790b0e6475fe887da00ede"} Oct 30 19:20:40.929627 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:40.929572 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" podStartSLOduration=1.929552489 podStartE2EDuration="1.929552489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:40.92825681 +0000 UTC m=+2800.955361772" watchObservedRunningTime="2026-10-30 19:20:40.929552489 +0000 UTC m=+2800.956657447" Oct 30 19:20:41.456953 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:41.456915 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:41.462780 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:41.462751 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:41.813200 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:41.813171 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:41.814236 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:41.814116 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:20:44.133766 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.133726 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll"] Oct 30 19:20:44.137662 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.137639 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.236092 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.235717 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-data\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.236534 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.236505 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qkc6\" (UniqueName: \"kubernetes.io/projected/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-kube-api-access-7qkc6\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.236662 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.236619 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istiod-ca-cert\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.236662 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.236652 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-workload-certs\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.236780 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.236692 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-podinfo\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.236849 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.236785 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-token\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.236849 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.236830 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-credential-socket\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.236957 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.236898 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-envoy\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.236957 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.236942 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-workload-socket\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.256154 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.255459 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-d4f84d-openshift-default-dockercfg-pl4gg\"" Oct 30 19:20:44.292872 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.292829 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll"] Oct 30 19:20:44.338105 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.338038 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-token\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.338105 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.338107 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-credential-socket\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.338413 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.338189 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-envoy\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.338413 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.338226 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-workload-socket\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.338413 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.338289 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-data\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.338592 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.338546 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-credential-socket\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.338651 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.338633 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-workload-socket\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.338711 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.338691 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qkc6\" (UniqueName: \"kubernetes.io/projected/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-kube-api-access-7qkc6\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.338775 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.338713 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-data\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.338775 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.338756 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istiod-ca-cert\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.338891 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.338785 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-workload-certs\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.338891 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.338822 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-podinfo\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.339309 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.339285 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-workload-certs\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.339503 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.339478 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istiod-ca-cert\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.341027 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.340997 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-envoy\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.341950 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.341917 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-podinfo\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.404199 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.400941 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qkc6\" (UniqueName: \"kubernetes.io/projected/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-kube-api-access-7qkc6\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.404199 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.403609 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-token\") pod \"e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.453508 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.453467 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:44.935697 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:20:44.935664 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea84ad8_53fd_4c3d_a32f_d202b3075ac1.slice/crio-d3d597aaed9140cf90329ee52e7a31a1df0864e2f5ed28bf9d798ee52ac61424 WatchSource:0}: Error finding container d3d597aaed9140cf90329ee52e7a31a1df0864e2f5ed28bf9d798ee52ac61424: Status 404 returned error can't find the container with id d3d597aaed9140cf90329ee52e7a31a1df0864e2f5ed28bf9d798ee52ac61424 Oct 30 19:20:44.936913 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.936888 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll"] Oct 30 19:20:44.938904 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.938865 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:20:44.938985 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.938939 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:20:44.938985 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:44.938965 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:20:45.842568 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:45.842527 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" event={"ID":"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1","Type":"ContainerStarted","Data":"9c555055604859de611acb6065076055f6996b57cb761f84f453b52621ee92be"} Oct 30 19:20:45.843018 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:45.842584 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" event={"ID":"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1","Type":"ContainerStarted","Data":"d3d597aaed9140cf90329ee52e7a31a1df0864e2f5ed28bf9d798ee52ac61424"} Oct 30 19:20:46.013598 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:46.013539 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" podStartSLOduration=2.013521869 podStartE2EDuration="2.013521869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:46.010844002 +0000 UTC m=+2806.037948963" watchObservedRunningTime="2026-10-30 19:20:46.013521869 +0000 UTC m=+2806.040626831" Oct 30 19:20:46.454188 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:46.454111 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:46.459944 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:46.459915 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:46.849278 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:46.849249 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:46.850222 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:46.850200 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:20:49.064779 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:49.064737 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90a0d000-tms7m"] Oct 30 19:20:49.113655 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:49.113618 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90a0d000-tms7m"] Oct 30 19:20:50.103961 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:50.103928 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09b855ed-4915-4eb3-9acf-46a3918dfda9" path="/var/lib/kubelet/pods/09b855ed-4915-4eb3-9acf-46a3918dfda9/volumes" Oct 30 19:20:50.371655 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:50.371569 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s"] Oct 30 19:20:50.375638 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:50.375613 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s" Oct 30 19:20:50.434801 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:50.434764 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-dc5cdd\"" Oct 30 19:20:50.484502 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:50.484466 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-dc5cdd-dockercfg-rndqt\"" Oct 30 19:20:50.488403 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:50.488368 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s"] Oct 30 19:20:50.491798 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:50.491768 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr"] Oct 30 19:20:50.495763 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:50.495737 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr" Oct 30 19:20:50.516710 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:50.516674 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0b354d-95e7-477f-b479-2028dcc5a950-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s\" (UID: \"cc0b354d-95e7-477f-b479-2028dcc5a950\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s" Oct 30 19:20:50.516928 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:50.516849 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxrwp\" (UniqueName: \"kubernetes.io/projected/cc0b354d-95e7-477f-b479-2028dcc5a950-kube-api-access-fxrwp\") pod \"payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s\" (UID: \"cc0b354d-95e7-477f-b479-2028dcc5a950\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s" Oct 30 19:20:50.550981 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:50.550945 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr"] Oct 30 19:20:50.625208 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:50.624188 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtllg\" (UniqueName: \"kubernetes.io/projected/eb431a4c-7eba-4bf5-ae56-56cba97e5342-kube-api-access-gtllg\") pod \"payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr\" (UID: \"eb431a4c-7eba-4bf5-ae56-56cba97e5342\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr" Oct 30 19:20:50.625208 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:50.624250 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxrwp\" (UniqueName: \"kubernetes.io/projected/cc0b354d-95e7-477f-b479-2028dcc5a950-kube-api-access-fxrwp\") pod \"payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s\" (UID: \"cc0b354d-95e7-477f-b479-2028dcc5a950\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s" Oct 30 19:20:50.625208 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:50.624386 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0b354d-95e7-477f-b479-2028dcc5a950-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s\" (UID: \"cc0b354d-95e7-477f-b479-2028dcc5a950\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s" Oct 30 19:20:50.625208 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:50.624465 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb431a4c-7eba-4bf5-ae56-56cba97e5342-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr\" (UID: \"eb431a4c-7eba-4bf5-ae56-56cba97e5342\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr" Oct 30 19:20:50.625754 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:50.625714 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0b354d-95e7-477f-b479-2028dcc5a950-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s\" (UID: \"cc0b354d-95e7-477f-b479-2028dcc5a950\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s" Oct 30 19:20:50.691563 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:50.691525 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxrwp\" (UniqueName: \"kubernetes.io/projected/cc0b354d-95e7-477f-b479-2028dcc5a950-kube-api-access-fxrwp\") pod \"payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s\" (UID: \"cc0b354d-95e7-477f-b479-2028dcc5a950\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s" Oct 30 19:20:50.725259 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:50.725215 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb431a4c-7eba-4bf5-ae56-56cba97e5342-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr\" (UID: \"eb431a4c-7eba-4bf5-ae56-56cba97e5342\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr" Oct 30 19:20:50.725479 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:50.725338 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtllg\" (UniqueName: \"kubernetes.io/projected/eb431a4c-7eba-4bf5-ae56-56cba97e5342-kube-api-access-gtllg\") pod \"payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr\" (UID: \"eb431a4c-7eba-4bf5-ae56-56cba97e5342\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr" Oct 30 19:20:50.725941 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:50.725920 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb431a4c-7eba-4bf5-ae56-56cba97e5342-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr\" (UID: \"eb431a4c-7eba-4bf5-ae56-56cba97e5342\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr" Oct 30 19:20:50.789728 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:50.789695 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtllg\" (UniqueName: \"kubernetes.io/projected/eb431a4c-7eba-4bf5-ae56-56cba97e5342-kube-api-access-gtllg\") pod \"payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr\" (UID: \"eb431a4c-7eba-4bf5-ae56-56cba97e5342\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr" Oct 30 19:20:50.810607 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:50.810568 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr" Oct 30 19:20:50.989427 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:50.989331 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s" Oct 30 19:20:51.205665 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:51.205631 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr"] Oct 30 19:20:51.206256 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:20:51.206227 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb431a4c_7eba_4bf5_ae56_56cba97e5342.slice/crio-f1b08390831e878cf8371ea68617119e7ce14fa67c34cd99aa2706395bd30630 WatchSource:0}: Error finding container f1b08390831e878cf8371ea68617119e7ce14fa67c34cd99aa2706395bd30630: Status 404 returned error can't find the container with id f1b08390831e878cf8371ea68617119e7ce14fa67c34cd99aa2706395bd30630 Oct 30 19:20:51.378292 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:51.378264 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s"] Oct 30 19:20:51.379415 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:20:51.379388 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0b354d_95e7_477f_b479_2028dcc5a950.slice/crio-f531a0d7514d0b0c693d40142352ef7217814fc6546f2b07c25b3e8da8755489 WatchSource:0}: Error finding container f531a0d7514d0b0c693d40142352ef7217814fc6546f2b07c25b3e8da8755489: Status 404 returned error can't find the container with id f531a0d7514d0b0c693d40142352ef7217814fc6546f2b07c25b3e8da8755489 Oct 30 19:20:51.897082 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:51.897041 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr" event={"ID":"eb431a4c-7eba-4bf5-ae56-56cba97e5342","Type":"ContainerStarted","Data":"7f2c676dd8a5494522d146d5888eb58c40baddfb46fee52feb28ccb1bee42b06"} Oct 30 19:20:51.897312 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:51.897102 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr" event={"ID":"eb431a4c-7eba-4bf5-ae56-56cba97e5342","Type":"ContainerStarted","Data":"f1b08390831e878cf8371ea68617119e7ce14fa67c34cd99aa2706395bd30630"} Oct 30 19:20:51.897312 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:51.897159 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr" Oct 30 19:20:51.898892 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:51.898867 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s" event={"ID":"cc0b354d-95e7-477f-b479-2028dcc5a950","Type":"ContainerStarted","Data":"a0510b235e5bc0cc1b090359f69c32afbaa15e1bff8a79cfeb48b09cc2840303"} Oct 30 19:20:51.898991 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:51.898912 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s" event={"ID":"cc0b354d-95e7-477f-b479-2028dcc5a950","Type":"ContainerStarted","Data":"f531a0d7514d0b0c693d40142352ef7217814fc6546f2b07c25b3e8da8755489"} Oct 30 19:20:51.898991 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:51.898939 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s" Oct 30 19:20:52.020288 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:52.020235 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr" podStartSLOduration=2.02021885 podStartE2EDuration="2.02021885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:52.01688326 +0000 UTC m=+2812.043988220" watchObservedRunningTime="2026-10-30 19:20:52.02021885 +0000 UTC m=+2812.047323808" Oct 30 19:20:52.135367 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:20:52.135312 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s" podStartSLOduration=2.135295661 podStartE2EDuration="2.135295661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:52.133609505 +0000 UTC m=+2812.160714464" watchObservedRunningTime="2026-10-30 19:20:52.135295661 +0000 UTC m=+2812.162400660" Oct 30 19:20:56.399390 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:20:56.399342 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:21:00.144752 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:00.144711 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ada8f68-fvbqp"] Oct 30 19:21:00.191751 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:00.191713 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ada8f68-fvbqp"] Oct 30 19:21:00.660371 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:00.660328 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7"] Oct 30 19:21:00.663979 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:00.663950 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7" Oct 30 19:21:00.717757 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:00.717720 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-f5dfb8\"" Oct 30 19:21:00.717957 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:00.717938 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-f5dfb8-dockercfg-fzq5d\"" Oct 30 19:21:00.732236 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:00.732194 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7"] Oct 30 19:21:00.751598 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:00.751546 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/865f93c1-d33c-4231-99ff-79bcf64acaca-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7\" (UID: \"865f93c1-d33c-4231-99ff-79bcf64acaca\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7" Oct 30 19:21:00.751806 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:00.751607 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rbvd\" (UniqueName: \"kubernetes.io/projected/865f93c1-d33c-4231-99ff-79bcf64acaca-kube-api-access-7rbvd\") pod \"payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7\" (UID: \"865f93c1-d33c-4231-99ff-79bcf64acaca\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7" Oct 30 19:21:00.829171 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:00.829094 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf"] Oct 30 19:21:00.835406 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:00.835310 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf" Oct 30 19:21:00.852796 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:00.852764 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kndqj\" (UniqueName: \"kubernetes.io/projected/bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890-kube-api-access-kndqj\") pod \"payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf\" (UID: \"bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf" Oct 30 19:21:00.852975 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:00.852943 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf\" (UID: \"bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf" Oct 30 19:21:00.853046 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:00.853027 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/865f93c1-d33c-4231-99ff-79bcf64acaca-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7\" (UID: \"865f93c1-d33c-4231-99ff-79bcf64acaca\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7" Oct 30 19:21:00.853092 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:00.853079 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rbvd\" (UniqueName: \"kubernetes.io/projected/865f93c1-d33c-4231-99ff-79bcf64acaca-kube-api-access-7rbvd\") pod \"payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7\" (UID: \"865f93c1-d33c-4231-99ff-79bcf64acaca\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7" Oct 30 19:21:00.855192 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:00.854539 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/865f93c1-d33c-4231-99ff-79bcf64acaca-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7\" (UID: \"865f93c1-d33c-4231-99ff-79bcf64acaca\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7" Oct 30 19:21:00.865242 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:00.865207 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf"] Oct 30 19:21:00.916480 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:00.916404 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rbvd\" (UniqueName: \"kubernetes.io/projected/865f93c1-d33c-4231-99ff-79bcf64acaca-kube-api-access-7rbvd\") pod \"payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7\" (UID: \"865f93c1-d33c-4231-99ff-79bcf64acaca\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7" Oct 30 19:21:00.954411 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:00.954375 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kndqj\" (UniqueName: \"kubernetes.io/projected/bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890-kube-api-access-kndqj\") pod \"payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf\" (UID: \"bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf" Oct 30 19:21:00.954582 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:00.954455 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf\" (UID: \"bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf" Oct 30 19:21:00.955118 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:00.955094 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf\" (UID: \"bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf" Oct 30 19:21:00.978004 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:00.977964 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7" Oct 30 19:21:01.017856 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:01.017820 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kndqj\" (UniqueName: \"kubernetes.io/projected/bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890-kube-api-access-kndqj\") pod \"payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf\" (UID: \"bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf" Oct 30 19:21:01.155566 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:01.155528 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf" Oct 30 19:21:01.366982 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:01.366947 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7"] Oct 30 19:21:01.367121 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:21:01.367009 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865f93c1_d33c_4231_99ff_79bcf64acaca.slice/crio-df37e00953069aad1a074a9b71f5e832006d1bacbd9f0e6631093c741ecb7ea9 WatchSource:0}: Error finding container df37e00953069aad1a074a9b71f5e832006d1bacbd9f0e6631093c741ecb7ea9: Status 404 returned error can't find the container with id df37e00953069aad1a074a9b71f5e832006d1bacbd9f0e6631093c741ecb7ea9 Oct 30 19:21:01.546974 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:01.546935 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf"] Oct 30 19:21:01.548112 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:21:01.548083 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb4d1d0b_5ca8_4cce_9e94_df6f2dd71890.slice/crio-0b11db2209218bad4e27898ea41ff6bbd792116e6c1637db29dd3ad51c6af991 WatchSource:0}: Error finding container 0b11db2209218bad4e27898ea41ff6bbd792116e6c1637db29dd3ad51c6af991: Status 404 returned error can't find the container with id 0b11db2209218bad4e27898ea41ff6bbd792116e6c1637db29dd3ad51c6af991 Oct 30 19:21:01.966925 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:01.966887 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf" event={"ID":"bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890","Type":"ContainerStarted","Data":"d2c268047fbbd9e02012b6d15aa92aa53f16d4de7e0f7df3b274bddca7678dde"} Oct 30 19:21:01.967156 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:01.966947 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf" event={"ID":"bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890","Type":"ContainerStarted","Data":"0b11db2209218bad4e27898ea41ff6bbd792116e6c1637db29dd3ad51c6af991"} Oct 30 19:21:01.967156 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:01.966984 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf" Oct 30 19:21:01.968790 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:01.968756 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7" event={"ID":"865f93c1-d33c-4231-99ff-79bcf64acaca","Type":"ContainerStarted","Data":"aca4c2e863728db8a0e7f43929b5435d8139adf3a0f03958e7917be5e4fa2936"} Oct 30 19:21:01.968883 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:01.968815 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7" event={"ID":"865f93c1-d33c-4231-99ff-79bcf64acaca","Type":"ContainerStarted","Data":"df37e00953069aad1a074a9b71f5e832006d1bacbd9f0e6631093c741ecb7ea9"} Oct 30 19:21:01.968883 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:01.968840 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7" Oct 30 19:21:02.088793 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:02.088730 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf" podStartSLOduration=2.088712277 podStartE2EDuration="2.088712277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:02.084471433 +0000 UTC m=+2822.111576393" watchObservedRunningTime="2026-10-30 19:21:02.088712277 +0000 UTC m=+2822.115817237" Oct 30 19:21:02.100267 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:02.100231 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8634059-e325-4cb7-95e2-a443a9ee8757" path="/var/lib/kubelet/pods/c8634059-e325-4cb7-95e2-a443a9ee8757/volumes" Oct 30 19:21:02.156322 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:02.156284 4531 scope.go:122] "RemoveContainer" containerID="9625943d87b168c88c650fb74519a2adfe0df6456ce58722ef3e1c0546d21f5d" Oct 30 19:21:02.170284 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:02.170254 4531 scope.go:122] "RemoveContainer" containerID="f3d1648489d1526df34dd9e2a236b77323c9d64eb91a4f7919be15dce7c74104" Oct 30 19:21:02.180454 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:02.180427 4531 scope.go:122] "RemoveContainer" containerID="8f244f10d01d1dadf4e3c3156a4c9303ca5e59803b69310dfe73e9759a1144c3" Oct 30 19:21:02.190213 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:02.190190 4531 scope.go:122] "RemoveContainer" containerID="5c9a9b1006649e886183d231cb44b9e56303652ed5071c7a43ddceb70bd7115d" Oct 30 19:21:02.908851 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:02.908815 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s" Oct 30 19:21:02.909078 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:02.908891 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr" Oct 30 19:21:03.028284 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:03.028222 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7" podStartSLOduration=3.028204005 podStartE2EDuration="3.028204005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:02.201462976 +0000 UTC m=+2822.228567940" watchObservedRunningTime="2026-10-30 19:21:03.028204005 +0000 UTC m=+2823.055308961" Oct 30 19:21:06.864618 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:06.864563 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn"] Oct 30 19:21:06.870889 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:06.870861 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn" Oct 30 19:21:06.927017 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:06.926984 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn"] Oct 30 19:21:06.929644 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:06.929611 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7962bfe1-7745-4603-90a1-246789d5a77d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn\" (UID: \"7962bfe1-7745-4603-90a1-246789d5a77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn" Oct 30 19:21:06.929800 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:06.929742 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7962bfe1-7745-4603-90a1-246789d5a77d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn\" (UID: \"7962bfe1-7745-4603-90a1-246789d5a77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn" Oct 30 19:21:06.929887 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:06.929835 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52zc7\" (UniqueName: \"kubernetes.io/projected/7962bfe1-7745-4603-90a1-246789d5a77d-kube-api-access-52zc7\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn\" (UID: \"7962bfe1-7745-4603-90a1-246789d5a77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn" Oct 30 19:21:06.971641 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:06.971606 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:21:06.971641 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:06.971604 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:21:06.971894 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:06.971622 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:21:06.971894 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:06.971660 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:21:07.031426 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:07.031388 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52zc7\" (UniqueName: \"kubernetes.io/projected/7962bfe1-7745-4603-90a1-246789d5a77d-kube-api-access-52zc7\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn\" (UID: \"7962bfe1-7745-4603-90a1-246789d5a77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn" Oct 30 19:21:07.031632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:07.031486 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7962bfe1-7745-4603-90a1-246789d5a77d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn\" (UID: \"7962bfe1-7745-4603-90a1-246789d5a77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn" Oct 30 19:21:07.031632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:07.031538 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7962bfe1-7745-4603-90a1-246789d5a77d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn\" (UID: \"7962bfe1-7745-4603-90a1-246789d5a77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn" Oct 30 19:21:07.032250 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:07.032227 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7962bfe1-7745-4603-90a1-246789d5a77d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn\" (UID: \"7962bfe1-7745-4603-90a1-246789d5a77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn" Oct 30 19:21:07.034281 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:07.034256 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7962bfe1-7745-4603-90a1-246789d5a77d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn\" (UID: \"7962bfe1-7745-4603-90a1-246789d5a77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn" Oct 30 19:21:07.093511 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:07.093476 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52zc7\" (UniqueName: \"kubernetes.io/projected/7962bfe1-7745-4603-90a1-246789d5a77d-kube-api-access-52zc7\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn\" (UID: \"7962bfe1-7745-4603-90a1-246789d5a77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn" Oct 30 19:21:07.186036 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:07.185936 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn" Oct 30 19:21:07.569955 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:07.569918 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn"] Oct 30 19:21:07.570698 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:21:07.570665 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7962bfe1_7745_4603_90a1_246789d5a77d.slice/crio-1eedbc822c8f2fcd96167bf3480dfb4db5a381f6460d2387dfffd7d3986436c1 WatchSource:0}: Error finding container 1eedbc822c8f2fcd96167bf3480dfb4db5a381f6460d2387dfffd7d3986436c1: Status 404 returned error can't find the container with id 1eedbc822c8f2fcd96167bf3480dfb4db5a381f6460d2387dfffd7d3986436c1 Oct 30 19:21:08.024576 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:08.024530 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn" event={"ID":"7962bfe1-7745-4603-90a1-246789d5a77d","Type":"ContainerStarted","Data":"4fcad77456f3b8b4ee0dfc146497f7db934027aca0fb9b572d0e11b581893140"} Oct 30 19:21:08.025007 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:08.024589 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn" event={"ID":"7962bfe1-7745-4603-90a1-246789d5a77d","Type":"ContainerStarted","Data":"1eedbc822c8f2fcd96167bf3480dfb4db5a381f6460d2387dfffd7d3986436c1"} Oct 30 19:21:08.140754 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:08.140635 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn" podStartSLOduration=2.140615216 podStartE2EDuration="2.140615216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:08.140280007 +0000 UTC m=+2828.167384969" watchObservedRunningTime="2026-10-30 19:21:08.140615216 +0000 UTC m=+2828.167720180" Oct 30 19:21:09.032422 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:09.032388 4531 generic.go:356] "Generic (PLEG): container finished" podID="7962bfe1-7745-4603-90a1-246789d5a77d" containerID="4fcad77456f3b8b4ee0dfc146497f7db934027aca0fb9b572d0e11b581893140" exitCode=0 Oct 30 19:21:09.032835 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:09.032479 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn" event={"ID":"7962bfe1-7745-4603-90a1-246789d5a77d","Type":"ContainerDied","Data":"4fcad77456f3b8b4ee0dfc146497f7db934027aca0fb9b572d0e11b581893140"} Oct 30 19:21:10.201265 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:10.201237 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn" Oct 30 19:21:10.272010 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:10.271969 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7962bfe1-7745-4603-90a1-246789d5a77d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7962bfe1-7745-4603-90a1-246789d5a77d-maas-api-openshift-ca\") pod \"7962bfe1-7745-4603-90a1-246789d5a77d\" (UID: \"7962bfe1-7745-4603-90a1-246789d5a77d\") " Oct 30 19:21:10.272228 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:10.272184 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7962bfe1-7745-4603-90a1-246789d5a77d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7962bfe1-7745-4603-90a1-246789d5a77d-maas-api-xks-ca\") pod \"7962bfe1-7745-4603-90a1-246789d5a77d\" (UID: \"7962bfe1-7745-4603-90a1-246789d5a77d\") " Oct 30 19:21:10.272279 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:10.272228 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7962bfe1-7745-4603-90a1-246789d5a77d-kube-api-access-52zc7\" (UniqueName: \"kubernetes.io/projected/7962bfe1-7745-4603-90a1-246789d5a77d-kube-api-access-52zc7\") pod \"7962bfe1-7745-4603-90a1-246789d5a77d\" (UID: \"7962bfe1-7745-4603-90a1-246789d5a77d\") " Oct 30 19:21:10.272418 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:10.272395 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7962bfe1-7745-4603-90a1-246789d5a77d-maas-api-openshift-ca" pod "7962bfe1-7745-4603-90a1-246789d5a77d" (UID: "7962bfe1-7745-4603-90a1-246789d5a77d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:21:10.272638 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:10.272621 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7962bfe1-7745-4603-90a1-246789d5a77d-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:21:10.274520 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:10.274493 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7962bfe1-7745-4603-90a1-246789d5a77d-maas-api-xks-ca" pod "7962bfe1-7745-4603-90a1-246789d5a77d" (UID: "7962bfe1-7745-4603-90a1-246789d5a77d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:21:10.274628 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:10.274604 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7962bfe1-7745-4603-90a1-246789d5a77d-kube-api-access-52zc7" pod "7962bfe1-7745-4603-90a1-246789d5a77d" (UID: "7962bfe1-7745-4603-90a1-246789d5a77d"). InnerVolumeSpecName "kube-api-access-52zc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:10.374194 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:10.374076 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7962bfe1-7745-4603-90a1-246789d5a77d-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:21:10.374194 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:10.374109 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52zc7\" (UniqueName: \"kubernetes.io/projected/7962bfe1-7745-4603-90a1-246789d5a77d-kube-api-access-52zc7\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:21:11.046857 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:11.046773 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn" Oct 30 19:21:11.046857 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:11.046808 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn" event={"ID":"7962bfe1-7745-4603-90a1-246789d5a77d","Type":"ContainerDied","Data":"1eedbc822c8f2fcd96167bf3480dfb4db5a381f6460d2387dfffd7d3986436c1"} Oct 30 19:21:11.047121 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:11.046863 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1eedbc822c8f2fcd96167bf3480dfb4db5a381f6460d2387dfffd7d3986436c1" Oct 30 19:21:12.979649 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:12.979613 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7" Oct 30 19:21:12.980169 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:12.979773 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf" Oct 30 19:21:16.085899 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:16.085859 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e7d337-8c827"] Oct 30 19:21:16.134665 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:16.134623 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e7d337-8c827"] Oct 30 19:21:18.101753 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:18.101710 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="587afca3-fc5e-4ddc-b0d3-d49713d4257d" path="/var/lib/kubelet/pods/587afca3-fc5e-4ddc-b0d3-d49713d4257d/volumes" Oct 30 19:21:22.814467 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:22.814429 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w"] Oct 30 19:21:22.814975 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:22.814959 4531 cpu_manager.go:414] "containerMap: removing container" podUID="7962bfe1-7745-4603-90a1-246789d5a77d" containerName="revoke-keys" Oct 30 19:21:22.815026 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:22.814976 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7962bfe1-7745-4603-90a1-246789d5a77d" containerName="revoke-keys" Oct 30 19:21:22.815065 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:22.815059 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7962bfe1-7745-4603-90a1-246789d5a77d" containerName="revoke-keys" Oct 30 19:21:22.820705 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:22.820671 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w" Oct 30 19:21:22.843073 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:22.843032 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxm68\" (UniqueName: \"kubernetes.io/projected/061d2407-1689-4ae4-b3c9-25d519671591-kube-api-access-xxm68\") pod \"payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w\" (UID: \"061d2407-1689-4ae4-b3c9-25d519671591\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w" Oct 30 19:21:22.843330 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:22.843211 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/061d2407-1689-4ae4-b3c9-25d519671591-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w\" (UID: \"061d2407-1689-4ae4-b3c9-25d519671591\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w" Oct 30 19:21:22.872773 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:22.872720 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-a56c7c\"" Oct 30 19:21:22.925778 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:22.925737 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w"] Oct 30 19:21:22.945008 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:22.944963 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxm68\" (UniqueName: \"kubernetes.io/projected/061d2407-1689-4ae4-b3c9-25d519671591-kube-api-access-xxm68\") pod \"payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w\" (UID: \"061d2407-1689-4ae4-b3c9-25d519671591\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w" Oct 30 19:21:22.945223 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:22.945081 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/061d2407-1689-4ae4-b3c9-25d519671591-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w\" (UID: \"061d2407-1689-4ae4-b3c9-25d519671591\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w" Oct 30 19:21:22.945866 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:22.945843 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/061d2407-1689-4ae4-b3c9-25d519671591-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w\" (UID: \"061d2407-1689-4ae4-b3c9-25d519671591\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w" Oct 30 19:21:23.020003 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:23.019962 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn"] Oct 30 19:21:23.024647 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:23.024554 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxm68\" (UniqueName: \"kubernetes.io/projected/061d2407-1689-4ae4-b3c9-25d519671591-kube-api-access-xxm68\") pod \"payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w\" (UID: \"061d2407-1689-4ae4-b3c9-25d519671591\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w" Oct 30 19:21:23.029209 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:23.029176 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn"] Oct 30 19:21:23.030474 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:23.030385 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn" Oct 30 19:21:23.047224 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:23.047181 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deda7597-7c2f-434d-83d0-a6f61c59cae9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn\" (UID: \"deda7597-7c2f-434d-83d0-a6f61c59cae9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn" Oct 30 19:21:23.047400 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:23.047341 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24bw7\" (UniqueName: \"kubernetes.io/projected/deda7597-7c2f-434d-83d0-a6f61c59cae9-kube-api-access-24bw7\") pod \"payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn\" (UID: \"deda7597-7c2f-434d-83d0-a6f61c59cae9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn" Oct 30 19:21:23.082804 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:23.082696 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-a56c7c-dockercfg-8cm7w\"" Oct 30 19:21:23.136380 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:23.136338 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w" Oct 30 19:21:23.148496 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:23.148458 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24bw7\" (UniqueName: \"kubernetes.io/projected/deda7597-7c2f-434d-83d0-a6f61c59cae9-kube-api-access-24bw7\") pod \"payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn\" (UID: \"deda7597-7c2f-434d-83d0-a6f61c59cae9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn" Oct 30 19:21:23.148683 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:23.148594 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deda7597-7c2f-434d-83d0-a6f61c59cae9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn\" (UID: \"deda7597-7c2f-434d-83d0-a6f61c59cae9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn" Oct 30 19:21:23.149339 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:23.149320 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deda7597-7c2f-434d-83d0-a6f61c59cae9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn\" (UID: \"deda7597-7c2f-434d-83d0-a6f61c59cae9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn" Oct 30 19:21:23.215729 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:23.214537 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24bw7\" (UniqueName: \"kubernetes.io/projected/deda7597-7c2f-434d-83d0-a6f61c59cae9-kube-api-access-24bw7\") pod \"payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn\" (UID: \"deda7597-7c2f-434d-83d0-a6f61c59cae9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn" Oct 30 19:21:23.352271 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:23.352173 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn" Oct 30 19:21:23.519488 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:23.519460 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w"] Oct 30 19:21:23.520746 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:21:23.520718 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061d2407_1689_4ae4_b3c9_25d519671591.slice/crio-4a0abd089f3bb90dc306baf43ef89881d180aa55c30f118bf7c3a3a257563f1c WatchSource:0}: Error finding container 4a0abd089f3bb90dc306baf43ef89881d180aa55c30f118bf7c3a3a257563f1c: Status 404 returned error can't find the container with id 4a0abd089f3bb90dc306baf43ef89881d180aa55c30f118bf7c3a3a257563f1c Oct 30 19:21:23.732746 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:23.732717 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn"] Oct 30 19:21:23.734557 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:21:23.734528 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeda7597_7c2f_434d_83d0_a6f61c59cae9.slice/crio-5164948a3bf2400d16a0f68844c481d873c9644abadab44e2a3bdcff86125d41 WatchSource:0}: Error finding container 5164948a3bf2400d16a0f68844c481d873c9644abadab44e2a3bdcff86125d41: Status 404 returned error can't find the container with id 5164948a3bf2400d16a0f68844c481d873c9644abadab44e2a3bdcff86125d41 Oct 30 19:21:24.140546 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:24.140496 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn" event={"ID":"deda7597-7c2f-434d-83d0-a6f61c59cae9","Type":"ContainerStarted","Data":"6eec38a8f73f5f2142786698238f62d9504f675787d34e3245de075783dabdb7"} Oct 30 19:21:24.141081 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:24.140575 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn" event={"ID":"deda7597-7c2f-434d-83d0-a6f61c59cae9","Type":"ContainerStarted","Data":"5164948a3bf2400d16a0f68844c481d873c9644abadab44e2a3bdcff86125d41"} Oct 30 19:21:24.141081 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:24.140602 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn" Oct 30 19:21:24.142532 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:24.142504 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w" event={"ID":"061d2407-1689-4ae4-b3c9-25d519671591","Type":"ContainerStarted","Data":"b6d2a173931f818d2525e05d7f54345897e61e19f5603e213dd0bfc4788be00a"} Oct 30 19:21:24.142665 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:24.142550 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w" event={"ID":"061d2407-1689-4ae4-b3c9-25d519671591","Type":"ContainerStarted","Data":"4a0abd089f3bb90dc306baf43ef89881d180aa55c30f118bf7c3a3a257563f1c"} Oct 30 19:21:24.142665 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:24.142644 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w" Oct 30 19:21:24.258182 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:24.257397 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn" podStartSLOduration=2.257369402 podStartE2EDuration="2.257369402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:24.25652825 +0000 UTC m=+2844.283633209" watchObservedRunningTime="2026-10-30 19:21:24.257369402 +0000 UTC m=+2844.284474361" Oct 30 19:21:24.377976 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:24.374834 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w" podStartSLOduration=2.374811587 podStartE2EDuration="2.374811587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:24.372507526 +0000 UTC m=+2844.399612497" watchObservedRunningTime="2026-10-30 19:21:24.374811587 +0000 UTC m=+2844.401916547" Oct 30 19:21:26.400180 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:21:26.400121 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:21:27.050413 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:27.050373 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07995457-g7p75"] Oct 30 19:21:27.099736 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:27.099694 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07995457-g7p75"] Oct 30 19:21:27.825527 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:27.825490 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn"] Oct 30 19:21:27.874188 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:27.874147 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d953c-65gxn"] Oct 30 19:21:28.098979 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:28.098894 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5600e823-769f-4162-8d78-9855c0c5f24b" path="/var/lib/kubelet/pods/5600e823-769f-4162-8d78-9855c0c5f24b/volumes" Oct 30 19:21:28.099371 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:28.099356 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7962bfe1-7745-4603-90a1-246789d5a77d" path="/var/lib/kubelet/pods/7962bfe1-7745-4603-90a1-246789d5a77d/volumes" Oct 30 19:21:35.153117 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:35.153082 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w" Oct 30 19:21:35.153678 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:35.153193 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn" Oct 30 19:21:36.234941 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:36.234898 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v"] Oct 30 19:21:36.239173 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:36.239122 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v" Oct 30 19:21:36.293543 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:36.293509 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:21:36.296427 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:36.296398 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:21:36.309475 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:36.309435 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwm7x\" (UniqueName: \"kubernetes.io/projected/c81e5540-acf8-45f4-a9a9-f99e38141e68-kube-api-access-lwm7x\") pod \"maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v\" (UID: \"c81e5540-acf8-45f4-a9a9-f99e38141e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v" Oct 30 19:21:36.309681 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:36.309562 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81e5540-acf8-45f4-a9a9-f99e38141e68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v\" (UID: \"c81e5540-acf8-45f4-a9a9-f99e38141e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v" Oct 30 19:21:36.309765 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:36.309695 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81e5540-acf8-45f4-a9a9-f99e38141e68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v\" (UID: \"c81e5540-acf8-45f4-a9a9-f99e38141e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v" Oct 30 19:21:36.341955 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:36.341911 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:21:36.342236 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:36.342019 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:21:36.346326 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:36.346291 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v"] Oct 30 19:21:36.410564 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:36.410521 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwm7x\" (UniqueName: \"kubernetes.io/projected/c81e5540-acf8-45f4-a9a9-f99e38141e68-kube-api-access-lwm7x\") pod \"maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v\" (UID: \"c81e5540-acf8-45f4-a9a9-f99e38141e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v" Oct 30 19:21:36.410755 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:36.410621 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81e5540-acf8-45f4-a9a9-f99e38141e68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v\" (UID: \"c81e5540-acf8-45f4-a9a9-f99e38141e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v" Oct 30 19:21:36.410755 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:36.410731 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81e5540-acf8-45f4-a9a9-f99e38141e68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v\" (UID: \"c81e5540-acf8-45f4-a9a9-f99e38141e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v" Oct 30 19:21:36.411485 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:36.411463 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81e5540-acf8-45f4-a9a9-f99e38141e68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v\" (UID: \"c81e5540-acf8-45f4-a9a9-f99e38141e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v" Oct 30 19:21:36.413456 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:36.413427 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81e5540-acf8-45f4-a9a9-f99e38141e68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v\" (UID: \"c81e5540-acf8-45f4-a9a9-f99e38141e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v" Oct 30 19:21:36.471943 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:36.471896 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwm7x\" (UniqueName: \"kubernetes.io/projected/c81e5540-acf8-45f4-a9a9-f99e38141e68-kube-api-access-lwm7x\") pod \"maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v\" (UID: \"c81e5540-acf8-45f4-a9a9-f99e38141e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v" Oct 30 19:21:36.553863 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:36.553832 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v" Oct 30 19:21:36.936365 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:36.936337 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v"] Oct 30 19:21:36.937643 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:21:36.937619 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81e5540_acf8_45f4_a9a9_f99e38141e68.slice/crio-9a1e665d9078437bcb5915cf68b8993764779892ad0b7055ed7f432e1f3a94e9 WatchSource:0}: Error finding container 9a1e665d9078437bcb5915cf68b8993764779892ad0b7055ed7f432e1f3a94e9: Status 404 returned error can't find the container with id 9a1e665d9078437bcb5915cf68b8993764779892ad0b7055ed7f432e1f3a94e9 Oct 30 19:21:37.231452 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:37.231425 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v" event={"ID":"c81e5540-acf8-45f4-a9a9-f99e38141e68","Type":"ContainerStarted","Data":"9a1e665d9078437bcb5915cf68b8993764779892ad0b7055ed7f432e1f3a94e9"} Oct 30 19:21:38.239638 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:38.239473 4531 generic.go:356] "Generic (PLEG): container finished" podID="c81e5540-acf8-45f4-a9a9-f99e38141e68" containerID="0fdd4bfa6a8356466d68a6cfd73d885ca6fed9622b9a32740f6f9280332e63c9" exitCode=0 Oct 30 19:21:38.239638 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:38.239557 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v" event={"ID":"c81e5540-acf8-45f4-a9a9-f99e38141e68","Type":"ContainerDied","Data":"0fdd4bfa6a8356466d68a6cfd73d885ca6fed9622b9a32740f6f9280332e63c9"} Oct 30 19:21:39.394068 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:39.394045 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v" Oct 30 19:21:39.545979 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:39.545879 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81e5540-acf8-45f4-a9a9-f99e38141e68-kube-api-access-lwm7x\" (UniqueName: \"kubernetes.io/projected/c81e5540-acf8-45f4-a9a9-f99e38141e68-kube-api-access-lwm7x\") pod \"c81e5540-acf8-45f4-a9a9-f99e38141e68\" (UID: \"c81e5540-acf8-45f4-a9a9-f99e38141e68\") " Oct 30 19:21:39.545979 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:39.545925 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c81e5540-acf8-45f4-a9a9-f99e38141e68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81e5540-acf8-45f4-a9a9-f99e38141e68-maas-api-openshift-ca\") pod \"c81e5540-acf8-45f4-a9a9-f99e38141e68\" (UID: \"c81e5540-acf8-45f4-a9a9-f99e38141e68\") " Oct 30 19:21:39.546295 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:39.546088 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c81e5540-acf8-45f4-a9a9-f99e38141e68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81e5540-acf8-45f4-a9a9-f99e38141e68-maas-api-xks-ca\") pod \"c81e5540-acf8-45f4-a9a9-f99e38141e68\" (UID: \"c81e5540-acf8-45f4-a9a9-f99e38141e68\") " Oct 30 19:21:39.546369 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:39.546318 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c81e5540-acf8-45f4-a9a9-f99e38141e68-maas-api-openshift-ca" pod "c81e5540-acf8-45f4-a9a9-f99e38141e68" (UID: "c81e5540-acf8-45f4-a9a9-f99e38141e68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:21:39.546603 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:39.546583 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81e5540-acf8-45f4-a9a9-f99e38141e68-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:21:39.548488 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:39.548450 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81e5540-acf8-45f4-a9a9-f99e38141e68-kube-api-access-lwm7x" pod "c81e5540-acf8-45f4-a9a9-f99e38141e68" (UID: "c81e5540-acf8-45f4-a9a9-f99e38141e68"). InnerVolumeSpecName "kube-api-access-lwm7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:39.548488 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:39.548458 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c81e5540-acf8-45f4-a9a9-f99e38141e68-maas-api-xks-ca" pod "c81e5540-acf8-45f4-a9a9-f99e38141e68" (UID: "c81e5540-acf8-45f4-a9a9-f99e38141e68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:21:39.648211 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:39.648167 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwm7x\" (UniqueName: \"kubernetes.io/projected/c81e5540-acf8-45f4-a9a9-f99e38141e68-kube-api-access-lwm7x\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:21:39.648211 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:39.648199 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81e5540-acf8-45f4-a9a9-f99e38141e68-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:21:40.254534 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:40.254502 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v" Oct 30 19:21:40.254737 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:40.254539 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v" event={"ID":"c81e5540-acf8-45f4-a9a9-f99e38141e68","Type":"ContainerDied","Data":"9a1e665d9078437bcb5915cf68b8993764779892ad0b7055ed7f432e1f3a94e9"} Oct 30 19:21:40.254737 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:40.254601 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a1e665d9078437bcb5915cf68b8993764779892ad0b7055ed7f432e1f3a94e9" Oct 30 19:21:43.050604 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:43.050569 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13672578-mw2ns"] Oct 30 19:21:43.091736 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:43.091706 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13672578-mw2ns"] Oct 30 19:21:44.098340 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:44.098300 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bf6f231-31d4-4dbc-a6ac-64d55521f6b3" path="/var/lib/kubelet/pods/6bf6f231-31d4-4dbc-a6ac-64d55521f6b3/volumes" Oct 30 19:21:51.058409 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:51.058372 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21ba810d-tm8xh"] Oct 30 19:21:51.107928 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:51.107894 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21ba810d-tm8xh"] Oct 30 19:21:52.098332 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:21:52.098295 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88eec190-49a8-4d52-ada9-9546abe31e3d" path="/var/lib/kubelet/pods/88eec190-49a8-4d52-ada9-9546abe31e3d/volumes" Oct 30 19:21:56.400369 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:21:56.400327 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:22:02.242725 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:02.242685 4531 scope.go:122] "RemoveContainer" containerID="26d072d07667690d9b81a524fb999f4d5d34994b80369c3208ee1e5a443b2acc" Oct 30 19:22:02.252794 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:02.252775 4531 scope.go:122] "RemoveContainer" containerID="1d8364e6e0e901b7467ac75f956092add540bf6602c979ecad8b87419abd93b3" Oct 30 19:22:02.262398 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:02.262376 4531 scope.go:122] "RemoveContainer" containerID="cd1d9395e47c54c58fdc906122ca1b835cb6e935366e0abec91b4475e00bfc03" Oct 30 19:22:02.275019 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:02.273008 4531 scope.go:122] "RemoveContainer" containerID="1e8602dc4b5911785e603f4a68bc4881ba042f07e1ec0388b1c62db849be36c8" Oct 30 19:22:06.051528 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:06.051492 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d3bff4b-dxzb7"] Oct 30 19:22:06.100966 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:06.100936 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d3bff4b-dxzb7"] Oct 30 19:22:08.098399 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:08.098369 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d22db1a8-7854-4889-a0f6-d2303262d880" path="/var/lib/kubelet/pods/d22db1a8-7854-4889-a0f6-d2303262d880/volumes" Oct 30 19:22:08.822049 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:08.822011 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v"] Oct 30 19:22:08.871642 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:08.871612 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9f939e-2z46v"] Oct 30 19:22:10.115785 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:10.115747 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c81e5540-acf8-45f4-a9a9-f99e38141e68" path="/var/lib/kubelet/pods/c81e5540-acf8-45f4-a9a9-f99e38141e68/volumes" Oct 30 19:22:12.063780 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:12.063725 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79495d21-zvcbg"] Oct 30 19:22:12.106458 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:12.106426 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79495d21-zvcbg"] Oct 30 19:22:14.099478 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:14.099435 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a38212e-aab1-4e15-abd6-23ed981aac8f" path="/var/lib/kubelet/pods/3a38212e-aab1-4e15-abd6-23ed981aac8f/volumes" Oct 30 19:22:17.891555 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:17.891501 4531 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:22:17.892329 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:17.892288 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" podUID="0ea84ad8-53fd-4c3d-a32f-d202b3075ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:22:20.543860 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:20.543820 4531 generic.go:356] "Generic (PLEG): container finished" podID="0ea84ad8-53fd-4c3d-a32f-d202b3075ac1" containerID="9c555055604859de611acb6065076055f6996b57cb761f84f453b52621ee92be" exitCode=0 Oct 30 19:22:20.544411 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:20.543890 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" event={"ID":"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1","Type":"ContainerDied","Data":"9c555055604859de611acb6065076055f6996b57cb761f84f453b52621ee92be"} Oct 30 19:22:20.544411 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:20.544369 4531 scope.go:122] "RemoveContainer" containerID="9c555055604859de611acb6065076055f6996b57cb761f84f453b52621ee92be" Oct 30 19:22:20.545757 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:20.545733 4531 generic.go:356] "Generic (PLEG): container finished" podID="0484c6c5-9f9e-408c-9974-a4f5f85b6bea" containerID="a41389ff7303cf57aa56f4ba914e97864d722a33b72ab90c2a940309552f90bb" exitCode=0 Oct 30 19:22:20.545864 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:20.545800 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" event={"ID":"0484c6c5-9f9e-408c-9974-a4f5f85b6bea","Type":"ContainerDied","Data":"a41389ff7303cf57aa56f4ba914e97864d722a33b72ab90c2a940309552f90bb"} Oct 30 19:22:20.546097 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:20.546068 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:22:20.546213 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:20.546160 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:22:20.546283 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:20.546212 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:22:20.546283 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:20.546215 4531 scope.go:122] "RemoveContainer" containerID="a41389ff7303cf57aa56f4ba914e97864d722a33b72ab90c2a940309552f90bb" Oct 30 19:22:20.547732 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:20.547701 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:22:20.547842 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:20.547771 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:22:20.547842 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:20.547806 4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:22:21.568985 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:21.568946 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" event={"ID":"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1","Type":"ContainerStarted","Data":"d67301a4f643826413b0828b0c2b559bf816956e0d8ee6c9a9a105bee53a0744"} Oct 30 19:22:21.571639 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:21.571611 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" event={"ID":"0484c6c5-9f9e-408c-9974-a4f5f85b6bea","Type":"ContainerStarted","Data":"41cfa381c5667829bb696c92ceee114df2d57b84febf90206e4b618b96d822e2"} Oct 30 19:22:21.667685 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:21.667647 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbb380da-487ts"] Oct 30 19:22:21.679156 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:21.674180 4531 cpu_manager.go:414] "containerMap: removing container" podUID="c81e5540-acf8-45f4-a9a9-f99e38141e68" containerName="revoke-keys" Oct 30 19:22:21.679156 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:21.678205 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c81e5540-acf8-45f4-a9a9-f99e38141e68" containerName="revoke-keys" Oct 30 19:22:21.679156 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:21.678479 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c81e5540-acf8-45f4-a9a9-f99e38141e68" containerName="revoke-keys" Oct 30 19:22:21.681257 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:21.681232 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbb380da-487ts" Oct 30 19:22:21.737155 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:21.736633 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:22:21.778485 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:21.777492 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:22:21.778485 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:21.777651 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:22:21.778485 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:21.778059 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:22:21.784800 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:21.784763 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbb380da-487ts"] Oct 30 19:22:21.889902 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:21.889809 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33307e75-c8a6-45a1-b6ab-959ffd695d99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbb380da-487ts\" (UID: \"33307e75-c8a6-45a1-b6ab-959ffd695d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbb380da-487ts" Oct 30 19:22:21.890280 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:21.890255 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc94d\" (UniqueName: \"kubernetes.io/projected/33307e75-c8a6-45a1-b6ab-959ffd695d99-kube-api-access-fc94d\") pod \"maas-api-revoke-sub-cbb380da-487ts\" (UID: \"33307e75-c8a6-45a1-b6ab-959ffd695d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbb380da-487ts" Oct 30 19:22:21.890488 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:21.890470 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33307e75-c8a6-45a1-b6ab-959ffd695d99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbb380da-487ts\" (UID: \"33307e75-c8a6-45a1-b6ab-959ffd695d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbb380da-487ts" Oct 30 19:22:21.991390 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:21.991345 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33307e75-c8a6-45a1-b6ab-959ffd695d99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbb380da-487ts\" (UID: \"33307e75-c8a6-45a1-b6ab-959ffd695d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbb380da-487ts" Oct 30 19:22:21.992263 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:21.992233 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33307e75-c8a6-45a1-b6ab-959ffd695d99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbb380da-487ts\" (UID: \"33307e75-c8a6-45a1-b6ab-959ffd695d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbb380da-487ts" Oct 30 19:22:21.992506 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:21.992493 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc94d\" (UniqueName: \"kubernetes.io/projected/33307e75-c8a6-45a1-b6ab-959ffd695d99-kube-api-access-fc94d\") pod \"maas-api-revoke-sub-cbb380da-487ts\" (UID: \"33307e75-c8a6-45a1-b6ab-959ffd695d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbb380da-487ts" Oct 30 19:22:21.992679 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:21.992361 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33307e75-c8a6-45a1-b6ab-959ffd695d99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbb380da-487ts\" (UID: \"33307e75-c8a6-45a1-b6ab-959ffd695d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbb380da-487ts" Oct 30 19:22:21.994936 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:21.994914 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33307e75-c8a6-45a1-b6ab-959ffd695d99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbb380da-487ts\" (UID: \"33307e75-c8a6-45a1-b6ab-959ffd695d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbb380da-487ts" Oct 30 19:22:22.068931 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:22.068860 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc94d\" (UniqueName: \"kubernetes.io/projected/33307e75-c8a6-45a1-b6ab-959ffd695d99-kube-api-access-fc94d\") pod \"maas-api-revoke-sub-cbb380da-487ts\" (UID: \"33307e75-c8a6-45a1-b6ab-959ffd695d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbb380da-487ts" Oct 30 19:22:22.298347 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:22.298316 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbb380da-487ts" Oct 30 19:22:22.455083 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:22.454281 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:22:22.455083 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:22.454727 4531 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 30 19:22:22.455083 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:22.454784 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" podUID="0ea84ad8-53fd-4c3d-a32f-d202b3075ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 30 19:22:22.471207 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:22.470479 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:22:22.471674 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:22.471562 4531 patch_prober.go:29] interesting pod/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 30 19:22:22.471674 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:22.471613 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" podUID="0484c6c5-9f9e-408c-9974-a4f5f85b6bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 30 19:22:22.768727 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:22:22.768689 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33307e75_c8a6_45a1_b6ab_959ffd695d99.slice/crio-249609291cdb024dcdd9a294de9f7650b858008342d182883076076e9d51572b WatchSource:0}: Error finding container 249609291cdb024dcdd9a294de9f7650b858008342d182883076076e9d51572b: Status 404 returned error can't find the container with id 249609291cdb024dcdd9a294de9f7650b858008342d182883076076e9d51572b Oct 30 19:22:22.769602 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:22.769393 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbb380da-487ts"] Oct 30 19:22:23.456709 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:23.456611 4531 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 30 19:22:23.456996 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:23.456949 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" podUID="0ea84ad8-53fd-4c3d-a32f-d202b3075ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 30 19:22:23.457786 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:23.457704 4531 patch_prober.go:29] interesting pod/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 30 19:22:23.457786 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:23.457743 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" podUID="0484c6c5-9f9e-408c-9974-a4f5f85b6bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 30 19:22:23.589396 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:23.589352 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbb380da-487ts" event={"ID":"33307e75-c8a6-45a1-b6ab-959ffd695d99","Type":"ContainerStarted","Data":"736eba678a20ccf0eb824f1f0a9381ee0ab268611a501a36c13c8a9d576b593d"} Oct 30 19:22:23.589672 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:23.589637 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbb380da-487ts" event={"ID":"33307e75-c8a6-45a1-b6ab-959ffd695d99","Type":"ContainerStarted","Data":"249609291cdb024dcdd9a294de9f7650b858008342d182883076076e9d51572b"} Oct 30 19:22:23.711502 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:23.711406 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbb380da-487ts" podStartSLOduration=2.711390391 podStartE2EDuration="2.711390391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:23.708277844 +0000 UTC m=+2903.735382803" watchObservedRunningTime="2026-10-30 19:22:23.711390391 +0000 UTC m=+2903.738495349" Oct 30 19:22:24.068706 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:24.068655 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d60811e6-4lcsf"] Oct 30 19:22:24.072924 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:24.072901 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d60811e6-4lcsf"] Oct 30 19:22:24.099119 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:24.099092 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="258738e9-9836-4b2a-a496-58d2183deb72" path="/var/lib/kubelet/pods/258738e9-9836-4b2a-a496-58d2183deb72/volumes" Oct 30 19:22:24.457212 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:24.456771 4531 patch_prober.go:29] interesting pod/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 30 19:22:24.457212 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:24.456847 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" podUID="0484c6c5-9f9e-408c-9974-a4f5f85b6bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 30 19:22:24.457483 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:24.457421 4531 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 30 19:22:24.457620 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:24.457496 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" podUID="0ea84ad8-53fd-4c3d-a32f-d202b3075ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 30 19:22:24.597491 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:24.597452 4531 generic.go:356] "Generic (PLEG): container finished" podID="33307e75-c8a6-45a1-b6ab-959ffd695d99" containerID="736eba678a20ccf0eb824f1f0a9381ee0ab268611a501a36c13c8a9d576b593d" exitCode=0 Oct 30 19:22:24.597663 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:24.597584 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbb380da-487ts" event={"ID":"33307e75-c8a6-45a1-b6ab-959ffd695d99","Type":"ContainerDied","Data":"736eba678a20ccf0eb824f1f0a9381ee0ab268611a501a36c13c8a9d576b593d"} Oct 30 19:22:25.458008 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:25.457543 4531 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 30 19:22:25.458008 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:25.457606 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" podUID="0ea84ad8-53fd-4c3d-a32f-d202b3075ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 30 19:22:25.458947 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:25.458923 4531 patch_prober.go:29] interesting pod/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 30 19:22:25.459106 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:25.459085 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" podUID="0484c6c5-9f9e-408c-9974-a4f5f85b6bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 30 19:22:25.895955 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:25.895933 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbb380da-487ts" Oct 30 19:22:25.983084 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:25.982999 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33307e75-c8a6-45a1-b6ab-959ffd695d99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33307e75-c8a6-45a1-b6ab-959ffd695d99-maas-api-xks-ca\") pod \"33307e75-c8a6-45a1-b6ab-959ffd695d99\" (UID: \"33307e75-c8a6-45a1-b6ab-959ffd695d99\") " Oct 30 19:22:25.983294 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:25.983089 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33307e75-c8a6-45a1-b6ab-959ffd695d99-kube-api-access-fc94d\" (UniqueName: \"kubernetes.io/projected/33307e75-c8a6-45a1-b6ab-959ffd695d99-kube-api-access-fc94d\") pod \"33307e75-c8a6-45a1-b6ab-959ffd695d99\" (UID: \"33307e75-c8a6-45a1-b6ab-959ffd695d99\") " Oct 30 19:22:25.983294 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:25.983224 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33307e75-c8a6-45a1-b6ab-959ffd695d99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33307e75-c8a6-45a1-b6ab-959ffd695d99-maas-api-openshift-ca\") pod \"33307e75-c8a6-45a1-b6ab-959ffd695d99\" (UID: \"33307e75-c8a6-45a1-b6ab-959ffd695d99\") " Oct 30 19:22:25.983679 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:25.983654 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33307e75-c8a6-45a1-b6ab-959ffd695d99-maas-api-openshift-ca" pod "33307e75-c8a6-45a1-b6ab-959ffd695d99" (UID: "33307e75-c8a6-45a1-b6ab-959ffd695d99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:22:25.985284 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:25.985259 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33307e75-c8a6-45a1-b6ab-959ffd695d99-maas-api-xks-ca" pod "33307e75-c8a6-45a1-b6ab-959ffd695d99" (UID: "33307e75-c8a6-45a1-b6ab-959ffd695d99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:22:25.985639 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:25.985619 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33307e75-c8a6-45a1-b6ab-959ffd695d99-kube-api-access-fc94d" pod "33307e75-c8a6-45a1-b6ab-959ffd695d99" (UID: "33307e75-c8a6-45a1-b6ab-959ffd695d99"). InnerVolumeSpecName "kube-api-access-fc94d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:26.084488 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:26.084454 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33307e75-c8a6-45a1-b6ab-959ffd695d99-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:22:26.084488 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:26.084482 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33307e75-c8a6-45a1-b6ab-959ffd695d99-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:22:26.084488 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:26.084492 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc94d\" (UniqueName: \"kubernetes.io/projected/33307e75-c8a6-45a1-b6ab-959ffd695d99-kube-api-access-fc94d\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:22:26.399788 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:22:26.399745 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:22:26.455564 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:26.455526 4531 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 30 19:22:26.455856 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:26.455829 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" podUID="0ea84ad8-53fd-4c3d-a32f-d202b3075ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 30 19:22:26.457564 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:26.457537 4531 patch_prober.go:29] interesting pod/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 30 19:22:26.457753 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:26.457731 4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" podUID="0484c6c5-9f9e-408c-9974-a4f5f85b6bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 30 19:22:26.614518 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:26.614487 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbb380da-487ts" Oct 30 19:22:26.614881 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:26.614524 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbb380da-487ts" event={"ID":"33307e75-c8a6-45a1-b6ab-959ffd695d99","Type":"ContainerDied","Data":"249609291cdb024dcdd9a294de9f7650b858008342d182883076076e9d51572b"} Oct 30 19:22:26.614881 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:26.614570 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="249609291cdb024dcdd9a294de9f7650b858008342d182883076076e9d51572b" Oct 30 19:22:26.813184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:26.813120 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:22:27.459144 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:27.459103 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:22:27.459450 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:27.459427 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:22:27.460159 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:27.460111 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:22:27.461794 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:27.461776 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:22:27.621985 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:27.621957 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:22:43.064098 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:43.064021 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74115f83-42g89"] Oct 30 19:22:43.087734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:43.087687 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74115f83-42g89"] Oct 30 19:22:44.098810 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:44.098772 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0832817-121a-44cb-b8a1-81c4446f6aea" path="/var/lib/kubelet/pods/f0832817-121a-44cb-b8a1-81c4446f6aea/volumes" Oct 30 19:22:56.399480 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:22:56.399432 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:22:57.057875 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:57.057822 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cca9728-jtdjp"] Oct 30 19:22:57.107123 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:57.107086 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cca9728-jtdjp"] Oct 30 19:22:58.098793 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:22:58.098756 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47d36714-4e59-4a7e-a853-59f2a40c4fdc" path="/var/lib/kubelet/pods/47d36714-4e59-4a7e-a853-59f2a40c4fdc/volumes" Oct 30 19:23:02.315682 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:02.315651 4531 scope.go:122] "RemoveContainer" containerID="eb27180946a5f28be49f5c4d9bbc369714c07c15ea8e3378ee5144d619060b9a" Oct 30 19:23:02.328519 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:02.328495 4531 scope.go:122] "RemoveContainer" containerID="1e1f944f324f8fe93e5abac6f30b729a7e50c357ec288620cdba341890611259" Oct 30 19:23:02.338030 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:02.338010 4531 scope.go:122] "RemoveContainer" containerID="42a32c3c6cac17901ba64ec728b37f7c4e7d646888024e67edc8efd19f9eaab3" Oct 30 19:23:02.347240 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:02.347222 4531 scope.go:122] "RemoveContainer" containerID="78a85a3088eb2fa38b050c39da9e29d5a6598873a0c86b5d01233a044fec1356" Oct 30 19:23:02.355441 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:02.355422 4531 scope.go:122] "RemoveContainer" containerID="b5dcda2adef101447573a0d827f15e5edf4910cef6e69ea796b62023a1926d04" Oct 30 19:23:02.363976 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:02.363958 4531 scope.go:122] "RemoveContainer" containerID="e92d8b69c7d1999443e2fc8e63eb6d3fb45cc933f75a295d31e0621874d142cc" Oct 30 19:23:05.911895 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:05.911853 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x"] Oct 30 19:23:05.912588 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:05.912566 4531 cpu_manager.go:414] "containerMap: removing container" podUID="33307e75-c8a6-45a1-b6ab-959ffd695d99" containerName="revoke-keys" Oct 30 19:23:05.912686 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:05.912591 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33307e75-c8a6-45a1-b6ab-959ffd695d99" containerName="revoke-keys" Oct 30 19:23:05.912751 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:05.912726 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="33307e75-c8a6-45a1-b6ab-959ffd695d99" containerName="revoke-keys" Oct 30 19:23:05.915967 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:05.915945 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" Oct 30 19:23:05.972219 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:05.972172 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-home\") pod \"auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x\" (UID: \"cd2ab45c-944c-4d14-8362-0831d16a2c59\") " pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" Oct 30 19:23:05.972421 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:05.972245 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-tmp-dir\") pod \"auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x\" (UID: \"cd2ab45c-944c-4d14-8362-0831d16a2c59\") " pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" Oct 30 19:23:05.972507 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:05.972459 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmmsl\" (UniqueName: \"kubernetes.io/projected/cd2ab45c-944c-4d14-8362-0831d16a2c59-kube-api-access-bmmsl\") pod \"auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x\" (UID: \"cd2ab45c-944c-4d14-8362-0831d16a2c59\") " pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" Oct 30 19:23:05.972574 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:05.972524 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-dshm\") pod \"auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x\" (UID: \"cd2ab45c-944c-4d14-8362-0831d16a2c59\") " pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" Oct 30 19:23:05.972626 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:05.972582 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd2ab45c-944c-4d14-8362-0831d16a2c59-tls-certs\") pod \"auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x\" (UID: \"cd2ab45c-944c-4d14-8362-0831d16a2c59\") " pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" Oct 30 19:23:05.973306 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:05.973276 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-model-cache\") pod \"auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x\" (UID: \"cd2ab45c-944c-4d14-8362-0831d16a2c59\") " pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" Oct 30 19:23:06.017652 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:06.017613 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-8f9c6b\"/\"kube-root-ca.crt\"" Oct 30 19:23:06.017857 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:06.017678 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-8f9c6b\"/\"openshift-service-ca.crt\"" Oct 30 19:23:06.017857 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:06.017700 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-8f9c6b\"/\"auth-test-model-8f9c6b-kserve-self-signed-certs\"" Oct 30 19:23:06.017857 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:06.017733 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-8f9c6b\"/\"default-dockercfg-wjlps\"" Oct 30 19:23:06.021177 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:06.021159 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x"] Oct 30 19:23:06.074756 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:06.074714 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-dshm\") pod \"auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x\" (UID: \"cd2ab45c-944c-4d14-8362-0831d16a2c59\") " pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" Oct 30 19:23:06.074756 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:06.074758 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd2ab45c-944c-4d14-8362-0831d16a2c59-tls-certs\") pod \"auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x\" (UID: \"cd2ab45c-944c-4d14-8362-0831d16a2c59\") " pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" Oct 30 19:23:06.075006 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:06.074828 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-model-cache\") pod \"auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x\" (UID: \"cd2ab45c-944c-4d14-8362-0831d16a2c59\") " pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" Oct 30 19:23:06.075006 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:06.074851 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-home\") pod \"auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x\" (UID: \"cd2ab45c-944c-4d14-8362-0831d16a2c59\") " pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" Oct 30 19:23:06.075006 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:06.074882 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-tmp-dir\") pod \"auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x\" (UID: \"cd2ab45c-944c-4d14-8362-0831d16a2c59\") " pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" Oct 30 19:23:06.075006 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:06.074932 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmmsl\" (UniqueName: \"kubernetes.io/projected/cd2ab45c-944c-4d14-8362-0831d16a2c59-kube-api-access-bmmsl\") pod \"auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x\" (UID: \"cd2ab45c-944c-4d14-8362-0831d16a2c59\") " pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" Oct 30 19:23:06.075279 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:06.075256 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-home\") pod \"auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x\" (UID: \"cd2ab45c-944c-4d14-8362-0831d16a2c59\") " pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" Oct 30 19:23:06.075343 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:06.075297 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-model-cache\") pod \"auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x\" (UID: \"cd2ab45c-944c-4d14-8362-0831d16a2c59\") " pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" Oct 30 19:23:06.075402 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:06.075351 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-tmp-dir\") pod \"auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x\" (UID: \"cd2ab45c-944c-4d14-8362-0831d16a2c59\") " pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" Oct 30 19:23:06.077307 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:06.077289 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-dshm\") pod \"auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x\" (UID: \"cd2ab45c-944c-4d14-8362-0831d16a2c59\") " pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" Oct 30 19:23:06.077584 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:06.077562 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd2ab45c-944c-4d14-8362-0831d16a2c59-tls-certs\") pod \"auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x\" (UID: \"cd2ab45c-944c-4d14-8362-0831d16a2c59\") " pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" Oct 30 19:23:06.144497 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:06.144460 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmmsl\" (UniqueName: \"kubernetes.io/projected/cd2ab45c-944c-4d14-8362-0831d16a2c59-kube-api-access-bmmsl\") pod \"auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x\" (UID: \"cd2ab45c-944c-4d14-8362-0831d16a2c59\") " pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" Oct 30 19:23:06.230921 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:06.230829 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" Oct 30 19:23:06.614705 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:06.614677 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x"] Oct 30 19:23:06.615360 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:23:06.615332 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd2ab45c_944c_4d14_8362_0831d16a2c59.slice/crio-6ce26e3b3b4b23987fc48ba54665bca17aefe421401bfad40d5821dfad8fdb38 WatchSource:0}: Error finding container 6ce26e3b3b4b23987fc48ba54665bca17aefe421401bfad40d5821dfad8fdb38: Status 404 returned error can't find the container with id 6ce26e3b3b4b23987fc48ba54665bca17aefe421401bfad40d5821dfad8fdb38 Oct 30 19:23:06.881012 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:06.880923 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" event={"ID":"cd2ab45c-944c-4d14-8362-0831d16a2c59","Type":"ContainerStarted","Data":"a2988b915d0f67cb24083a09774c47df58fc2aa49ee3776fd45033b8345ffdca"} Oct 30 19:23:06.881012 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:06.880964 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" event={"ID":"cd2ab45c-944c-4d14-8362-0831d16a2c59","Type":"ContainerStarted","Data":"6ce26e3b3b4b23987fc48ba54665bca17aefe421401bfad40d5821dfad8fdb38"} Oct 30 19:23:06.999505 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:06.999455 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" podStartSLOduration=1.999439433 podStartE2EDuration="1.999439433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:06.997635834 +0000 UTC m=+2947.024740793" watchObservedRunningTime="2026-10-30 19:23:06.999439433 +0000 UTC m=+2947.026544392" Oct 30 19:23:07.053372 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:07.053328 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-253a5641-2fn7r"] Oct 30 19:23:07.056048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:07.056026 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-253a5641-2fn7r"] Oct 30 19:23:07.231003 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:07.230921 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" Oct 30 19:23:08.098545 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:08.098510 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d83c04f6-4c26-4b1f-916a-7ebfee306cbd" path="/var/lib/kubelet/pods/d83c04f6-4c26-4b1f-916a-7ebfee306cbd/volumes" Oct 30 19:23:13.061906 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:13.061867 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70c1191d-wlzjh"] Oct 30 19:23:13.111414 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:13.111377 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70c1191d-wlzjh"] Oct 30 19:23:14.053803 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:14.053770 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a21902d6-npxp7"] Oct 30 19:23:14.098492 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:14.098461 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="106dc01f-378d-4b71-8da7-dbae2d87a0a8" path="/var/lib/kubelet/pods/106dc01f-378d-4b71-8da7-dbae2d87a0a8/volumes" Oct 30 19:23:14.103279 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:14.103220 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a21902d6-npxp7"] Oct 30 19:23:16.099559 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:16.099519 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef8bd865-8f5f-4bb0-aa15-69d8cd946642" path="/var/lib/kubelet/pods/ef8bd865-8f5f-4bb0-aa15-69d8cd946642/volumes" Oct 30 19:23:16.231036 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:16.230994 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" Oct 30 19:23:16.244113 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:16.244076 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" Oct 30 19:23:16.965862 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:16.965833 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" Oct 30 19:23:20.068445 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:20.068415 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a24f4dc-25blv"] Oct 30 19:23:20.117941 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:20.117909 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a24f4dc-25blv"] Oct 30 19:23:20.146052 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:23:20.146018 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-2a24f4dc-25blv\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="93a22968-0fc8-4713-b803-6c42a162192d" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24f4dc-25blv" Oct 30 19:23:22.098536 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:22.098500 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93a22968-0fc8-4713-b803-6c42a162192d" path="/var/lib/kubelet/pods/93a22968-0fc8-4713-b803-6c42a162192d/volumes" Oct 30 19:23:26.404262 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:23:26.404223 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:23:37.066351 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:37.066309 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-226e0e2b-p2zxs"] Oct 30 19:23:37.071632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:37.071606 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92a026c3-prpbp"] Oct 30 19:23:37.073762 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:37.073743 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-226e0e2b-p2zxs"] Oct 30 19:23:37.116018 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:37.115979 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92a026c3-prpbp"] Oct 30 19:23:38.065051 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:38.065013 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa70584f-fkhfl"] Oct 30 19:23:38.085708 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:38.085670 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa70584f-fkhfl"] Oct 30 19:23:38.098346 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:38.098322 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4976184c-ddb4-4a68-8af2-27e0a316a1df" path="/var/lib/kubelet/pods/4976184c-ddb4-4a68-8af2-27e0a316a1df/volumes" Oct 30 19:23:38.098760 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:38.098742 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5" path="/var/lib/kubelet/pods/8e0ce9b3-caab-4d5d-8b04-a5b3ccc89fb5/volumes" Oct 30 19:23:38.099140 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:38.099117 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a97b9dea-b1ce-4818-931a-7060a9562bae" path="/var/lib/kubelet/pods/a97b9dea-b1ce-4818-931a-7060a9562bae/volumes" Oct 30 19:23:48.059759 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:48.059719 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3500cb0-jbxdh"] Oct 30 19:23:48.109324 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:48.109289 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3500cb0-jbxdh"] Oct 30 19:23:50.098613 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:23:50.098579 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c10d4e49-7a07-49ea-9707-66956ddd6ceb" path="/var/lib/kubelet/pods/c10d4e49-7a07-49ea-9707-66956ddd6ceb/volumes" Oct 30 19:23:56.399442 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:23:56.399394 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:24:02.410334 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:02.410298 4531 scope.go:122] "RemoveContainer" containerID="a4e5d8a1af656470020cb36ad248896419c7a3c46a541ddb874751fd5e95ea8c" Oct 30 19:24:02.421033 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:02.421007 4531 scope.go:122] "RemoveContainer" containerID="fff0965ee53b9e07bf6eb60b16f719f5cc0c4a4bd2eada1b242ad8cc98029805" Oct 30 19:24:02.430731 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:02.430707 4531 scope.go:122] "RemoveContainer" containerID="6a88d01225918396c0c70666b6b738dcd0a25ed1ce7901bca3bcb37eb99143b2" Oct 30 19:24:02.440211 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:02.440182 4531 scope.go:122] "RemoveContainer" containerID="954cccde6c438f7fad55efcc734bb71cdc90ec3e3feba1ee8875ac0a60c63009" Oct 30 19:24:02.449165 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:02.449123 4531 scope.go:122] "RemoveContainer" containerID="5b9c8f25496b002ed36b413fe2f0a5791b1cae13632229907db4274861846c66" Oct 30 19:24:02.459679 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:02.459516 4531 scope.go:122] "RemoveContainer" containerID="906de970a81b1d66943fd4141e91f76c2eb4cdb6d9e841ba24c3344e5a113d9d" Oct 30 19:24:02.470999 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:02.470979 4531 scope.go:122] "RemoveContainer" containerID="b833ab47d1fb8c29a198f16d945adebcd3c313ba7d3763e4c2eb4730ab399cc0" Oct 30 19:24:26.400000 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:24:26.399962 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:24:27.495589 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:27.495547 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh"] Oct 30 19:24:27.496008 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:27.495800 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" podUID="5987b871-3251-4621-b79e-198239ddce66" containerName="istio-proxy" containerID="cri-o://90934e5b573a94e33eda67ec31f9e64e4c678ba53e7b5be3f072376d9cb11ebd" gracePeriod=30 Oct 30 19:24:32.775964 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:32.775897 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:24:32.918265 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:32.918235 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5987b871-3251-4621-b79e-198239ddce66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5987b871-3251-4621-b79e-198239ddce66-istio-podinfo\") pod \"5987b871-3251-4621-b79e-198239ddce66\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " Oct 30 19:24:32.918469 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:32.918287 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-credential-socket\") pod \"5987b871-3251-4621-b79e-198239ddce66\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " Oct 30 19:24:32.918469 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:32.918330 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-workload-socket\") pod \"5987b871-3251-4621-b79e-198239ddce66\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " Oct 30 19:24:32.918469 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:32.918383 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-istio-data\") pod \"5987b871-3251-4621-b79e-198239ddce66\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " Oct 30 19:24:32.918469 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:32.918423 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-workload-certs\") pod \"5987b871-3251-4621-b79e-198239ddce66\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " Oct 30 19:24:32.918469 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:32.918441 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-istio-envoy\") pod \"5987b871-3251-4621-b79e-198239ddce66\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " Oct 30 19:24:32.918753 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:32.918484 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5987b871-3251-4621-b79e-198239ddce66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5987b871-3251-4621-b79e-198239ddce66-istiod-ca-cert\") pod \"5987b871-3251-4621-b79e-198239ddce66\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " Oct 30 19:24:32.918753 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:32.918513 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5987b871-3251-4621-b79e-198239ddce66-kube-api-access-2c4f7\" (UniqueName: \"kubernetes.io/projected/5987b871-3251-4621-b79e-198239ddce66-kube-api-access-2c4f7\") pod \"5987b871-3251-4621-b79e-198239ddce66\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " Oct 30 19:24:32.918753 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:32.918601 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-workload-socket" pod "5987b871-3251-4621-b79e-198239ddce66" (UID: "5987b871-3251-4621-b79e-198239ddce66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:24:32.918753 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:32.918618 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-credential-socket" pod "5987b871-3251-4621-b79e-198239ddce66" (UID: "5987b871-3251-4621-b79e-198239ddce66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:24:32.918753 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:32.918682 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-workload-certs" pod "5987b871-3251-4621-b79e-198239ddce66" (UID: "5987b871-3251-4621-b79e-198239ddce66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:24:32.918753 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:32.918687 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5987b871-3251-4621-b79e-198239ddce66-istio-token\" (UniqueName: \"kubernetes.io/projected/5987b871-3251-4621-b79e-198239ddce66-istio-token\") pod \"5987b871-3251-4621-b79e-198239ddce66\" (UID: \"5987b871-3251-4621-b79e-198239ddce66\") " Oct 30 19:24:32.919065 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:32.918918 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5987b871-3251-4621-b79e-198239ddce66-istiod-ca-cert" pod "5987b871-3251-4621-b79e-198239ddce66" (UID: "5987b871-3251-4621-b79e-198239ddce66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:24:32.919281 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:32.919255 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-istio-data" pod "5987b871-3251-4621-b79e-198239ddce66" (UID: "5987b871-3251-4621-b79e-198239ddce66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:24:32.919628 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:32.919523 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:24:32.919763 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:32.919742 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5987b871-3251-4621-b79e-198239ddce66-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:24:32.919854 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:32.919771 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:24:32.919854 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:32.919789 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:24:32.921012 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:32.920992 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5987b871-3251-4621-b79e-198239ddce66-istio-token" pod "5987b871-3251-4621-b79e-198239ddce66" (UID: "5987b871-3251-4621-b79e-198239ddce66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:24:32.921119 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:32.921017 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5987b871-3251-4621-b79e-198239ddce66-kube-api-access-2c4f7" pod "5987b871-3251-4621-b79e-198239ddce66" (UID: "5987b871-3251-4621-b79e-198239ddce66"). InnerVolumeSpecName "kube-api-access-2c4f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:24:32.921205 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:32.921190 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5987b871-3251-4621-b79e-198239ddce66-istio-podinfo" pod "5987b871-3251-4621-b79e-198239ddce66" (UID: "5987b871-3251-4621-b79e-198239ddce66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:24:32.921249 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:32.921216 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-istio-envoy" pod "5987b871-3251-4621-b79e-198239ddce66" (UID: "5987b871-3251-4621-b79e-198239ddce66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:24:33.020485 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:33.020443 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c4f7\" (UniqueName: \"kubernetes.io/projected/5987b871-3251-4621-b79e-198239ddce66-kube-api-access-2c4f7\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:24:33.020485 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:33.020472 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5987b871-3251-4621-b79e-198239ddce66-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:24:33.020485 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:33.020481 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5987b871-3251-4621-b79e-198239ddce66-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:24:33.020485 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:33.020490 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:24:33.020849 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:33.020498 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5987b871-3251-4621-b79e-198239ddce66-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:24:33.454067 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:33.454024 4531 generic.go:356] "Generic (PLEG): container finished" podID="5987b871-3251-4621-b79e-198239ddce66" containerID="90934e5b573a94e33eda67ec31f9e64e4c678ba53e7b5be3f072376d9cb11ebd" exitCode=0 Oct 30 19:24:33.454395 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:33.454077 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" Oct 30 19:24:33.454395 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:33.454113 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" event={"ID":"5987b871-3251-4621-b79e-198239ddce66","Type":"ContainerDied","Data":"90934e5b573a94e33eda67ec31f9e64e4c678ba53e7b5be3f072376d9cb11ebd"} Oct 30 19:24:33.454395 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:33.454207 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh" event={"ID":"5987b871-3251-4621-b79e-198239ddce66","Type":"ContainerDied","Data":"9d917bb0b27d493f319ba885e28f046087b62d15538192cdedb2e55108f9b42b"} Oct 30 19:24:33.454395 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:33.454240 4531 scope.go:122] "RemoveContainer" containerID="90934e5b573a94e33eda67ec31f9e64e4c678ba53e7b5be3f072376d9cb11ebd" Oct 30 19:24:33.467840 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:33.467796 4531 scope.go:122] "RemoveContainer" containerID="90934e5b573a94e33eda67ec31f9e64e4c678ba53e7b5be3f072376d9cb11ebd" Oct 30 19:24:33.468354 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:24:33.468283 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90934e5b573a94e33eda67ec31f9e64e4c678ba53e7b5be3f072376d9cb11ebd\": container with ID starting with 90934e5b573a94e33eda67ec31f9e64e4c678ba53e7b5be3f072376d9cb11ebd not found: ID does not exist" containerID="90934e5b573a94e33eda67ec31f9e64e4c678ba53e7b5be3f072376d9cb11ebd" Oct 30 19:24:33.468354 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:33.468320 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90934e5b573a94e33eda67ec31f9e64e4c678ba53e7b5be3f072376d9cb11ebd"} err="failed to get container status \"90934e5b573a94e33eda67ec31f9e64e4c678ba53e7b5be3f072376d9cb11ebd\": rpc error: code = NotFound desc = could not find container \"90934e5b573a94e33eda67ec31f9e64e4c678ba53e7b5be3f072376d9cb11ebd\": container with ID starting with 90934e5b573a94e33eda67ec31f9e64e4c678ba53e7b5be3f072376d9cb11ebd not found: ID does not exist" Oct 30 19:24:33.641979 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:33.641944 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh"] Oct 30 19:24:33.691179 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:33.691145 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-mktfh"] Oct 30 19:24:34.099177 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:24:34.099120 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5987b871-3251-4621-b79e-198239ddce66" path="/var/lib/kubelet/pods/5987b871-3251-4621-b79e-198239ddce66/volumes" Oct 30 19:24:56.397341 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:24:56.397304 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:25:02.524110 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:25:02.524072 4531 scope.go:122] "RemoveContainer" containerID="468dd05d379d653b3c6d8110e5a070d6358dfef7e351cb46575c6c963c6dfcdc" Oct 30 19:25:02.534503 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:25:02.534465 4531 scope.go:122] "RemoveContainer" containerID="d81766ce9b60f917810aa717508324a0d3f9bfdb99f4f2d5aff0896bda3663d6" Oct 30 19:25:02.547411 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:25:02.547362 4531 scope.go:122] "RemoveContainer" containerID="b9cdbe3269ba6252b6d7f236dd95fb6a8017b2b112a4499432f61aa86bf0442c" Oct 30 19:25:02.558213 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:25:02.558186 4531 scope.go:122] "RemoveContainer" containerID="207cbefc64277798c8a9578ad4f2e2459e210c79209eca938f196e42c5b2eee1" Oct 30 19:25:02.568892 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:25:02.568861 4531 scope.go:122] "RemoveContainer" containerID="18396c05cf209c16aebac1eb57311ce60aee44d0e3563345407af645e5244533" Oct 30 19:25:02.579491 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:25:02.579467 4531 scope.go:122] "RemoveContainer" containerID="7004cd34afdbea6ebb9f865228f416c52366b3d037963288c98ea2aac60b3026" Oct 30 19:25:02.590103 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:25:02.590077 4531 scope.go:122] "RemoveContainer" containerID="351e7f20a283827505a0194c1ca92224283e43a343da5982255be5914333a104" Oct 30 19:25:02.601281 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:25:02.601257 4531 scope.go:122] "RemoveContainer" containerID="f7b1fefe460c70baa949b039fd1eb732f9c58ba912b4a7fd1034ff41c3d4d4f9" Oct 30 19:25:02.611800 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:25:02.611776 4531 scope.go:122] "RemoveContainer" containerID="8df31c6fe6cc732006de2c604233a83d229105a1103297275537500dc35d5c16" Oct 30 19:25:26.400501 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:25:26.400458 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:25:56.402631 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:25:56.402590 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:26:13.331612 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:13.331565 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb"] Oct 30 19:26:13.332236 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:13.331796 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" podUID="f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4" containerName="istio-proxy" containerID="cri-o://7f2d5aae037dbdc4490458e36b63877d849c855fbc0c9f7f6087baeed5addb42" gracePeriod=30 Oct 30 19:26:17.296195 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:17.296150 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:26:17.296636 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:17.296228 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" podUID="f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:26:18.618713 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.618635 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:26:18.658490 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.658445 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-data\") pod \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " Oct 30 19:26:18.658687 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.658513 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-credential-socket\") pod \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " Oct 30 19:26:18.658687 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.658606 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-workload-socket\") pod \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " Oct 30 19:26:18.658687 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.658631 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-podinfo\") pod \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " Oct 30 19:26:18.658687 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.658666 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-kube-api-access-9sw7n\" (UniqueName: \"kubernetes.io/projected/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-kube-api-access-9sw7n\") pod \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " Oct 30 19:26:18.658936 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.658721 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istiod-ca-cert\") pod \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " Oct 30 19:26:18.658936 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.658780 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-token\" (UniqueName: \"kubernetes.io/projected/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-token\") pod \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " Oct 30 19:26:18.658936 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.658829 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-credential-socket" pod "f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4" (UID: "f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:26:18.658936 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.658846 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-data" pod "f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4" (UID: "f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:26:18.658936 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.658871 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-workload-certs\") pod \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " Oct 30 19:26:18.659238 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.658952 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-workload-socket" pod "f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4" (UID: "f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:26:18.659238 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.659006 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-envoy\") pod \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\" (UID: \"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4\") " Oct 30 19:26:18.659238 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.659083 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-workload-certs" pod "f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4" (UID: "f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:26:18.659372 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.659269 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istiod-ca-cert" pod "f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4" (UID: "f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:26:18.659787 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.659759 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:26:18.659787 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.659788 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:26:18.659990 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.659804 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:26:18.659990 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.659822 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:26:18.659990 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.659836 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:26:18.661295 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.661262 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-podinfo" pod "f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4" (UID: "f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:26:18.661423 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.661320 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-envoy" pod "f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4" (UID: "f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:26:18.661423 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.661326 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-kube-api-access-9sw7n" pod "f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4" (UID: "f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4"). InnerVolumeSpecName "kube-api-access-9sw7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:26:18.661524 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.661428 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-token" pod "f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4" (UID: "f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:26:18.761113 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.761073 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:26:18.761113 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.761110 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:26:18.761113 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.761120 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:26:18.761546 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:18.761146 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sw7n\" (UniqueName: \"kubernetes.io/projected/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4-kube-api-access-9sw7n\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:26:19.109943 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:19.109898 4531 generic.go:356] "Generic (PLEG): container finished" podID="f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4" containerID="7f2d5aae037dbdc4490458e36b63877d849c855fbc0c9f7f6087baeed5addb42" exitCode=0 Oct 30 19:26:19.110213 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:19.109962 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" Oct 30 19:26:19.110213 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:19.109986 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" event={"ID":"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4","Type":"ContainerDied","Data":"7f2d5aae037dbdc4490458e36b63877d849c855fbc0c9f7f6087baeed5addb42"} Oct 30 19:26:19.110213 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:19.110048 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb" event={"ID":"f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4","Type":"ContainerDied","Data":"2da6b1f176901b28473f382f1f9d7b3fe4fee7ea291cbd0f214696e99454db23"} Oct 30 19:26:19.110213 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:19.110073 4531 scope.go:122] "RemoveContainer" containerID="7f2d5aae037dbdc4490458e36b63877d849c855fbc0c9f7f6087baeed5addb42" Oct 30 19:26:19.121541 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:19.121508 4531 scope.go:122] "RemoveContainer" containerID="7f2d5aae037dbdc4490458e36b63877d849c855fbc0c9f7f6087baeed5addb42" Oct 30 19:26:19.121897 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:26:19.121872 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f2d5aae037dbdc4490458e36b63877d849c855fbc0c9f7f6087baeed5addb42\": container with ID starting with 7f2d5aae037dbdc4490458e36b63877d849c855fbc0c9f7f6087baeed5addb42 not found: ID does not exist" containerID="7f2d5aae037dbdc4490458e36b63877d849c855fbc0c9f7f6087baeed5addb42" Oct 30 19:26:19.121988 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:19.121906 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f2d5aae037dbdc4490458e36b63877d849c855fbc0c9f7f6087baeed5addb42"} err="failed to get container status \"7f2d5aae037dbdc4490458e36b63877d849c855fbc0c9f7f6087baeed5addb42\": rpc error: code = NotFound desc = could not find container \"7f2d5aae037dbdc4490458e36b63877d849c855fbc0c9f7f6087baeed5addb42\": container with ID starting with 7f2d5aae037dbdc4490458e36b63877d849c855fbc0c9f7f6087baeed5addb42 not found: ID does not exist" Oct 30 19:26:19.276030 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:19.275984 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb"] Oct 30 19:26:19.325967 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:19.325923 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-tz8zb"] Oct 30 19:26:20.101013 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:20.100974 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4" path="/var/lib/kubelet/pods/f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4/volumes" Oct 30 19:26:26.399749 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:26:26.399712 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:26:29.117052 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.117010 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr"] Oct 30 19:26:29.117871 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.117852 4531 cpu_manager.go:414] "containerMap: removing container" podUID="f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4" containerName="istio-proxy" Oct 30 19:26:29.117926 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.117874 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4" containerName="istio-proxy" Oct 30 19:26:29.117926 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.117889 4531 cpu_manager.go:414] "containerMap: removing container" podUID="5987b871-3251-4621-b79e-198239ddce66" containerName="istio-proxy" Oct 30 19:26:29.117926 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.117897 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5987b871-3251-4621-b79e-198239ddce66" containerName="istio-proxy" Oct 30 19:26:29.118038 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.118026 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f93ef69c-5501-4a9f-a9f8-9b3b1c8ab1e4" containerName="istio-proxy" Oct 30 19:26:29.118075 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.118042 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5987b871-3251-4621-b79e-198239ddce66" containerName="istio-proxy" Oct 30 19:26:29.122885 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.122861 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" Oct 30 19:26:29.176142 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.176094 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-c60f5c\"/\"openshift-service-ca.crt\"" Oct 30 19:26:29.176308 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.176160 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-c60f5c\"/\"kube-root-ca.crt\"" Oct 30 19:26:29.187954 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.187908 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-dshm\") pod \"autores-model-c60f5c-kserve-5777c77469-hcrrr\" (UID: \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\") " pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" Oct 30 19:26:29.188122 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.187958 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-model-cache\") pod \"autores-model-c60f5c-kserve-5777c77469-hcrrr\" (UID: \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\") " pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" Oct 30 19:26:29.188122 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.188049 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-home\") pod \"autores-model-c60f5c-kserve-5777c77469-hcrrr\" (UID: \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\") " pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" Oct 30 19:26:29.188273 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.188120 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f787f\" (UniqueName: \"kubernetes.io/projected/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-kube-api-access-f787f\") pod \"autores-model-c60f5c-kserve-5777c77469-hcrrr\" (UID: \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\") " pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" Oct 30 19:26:29.188311 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.188277 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-tmp-dir\") pod \"autores-model-c60f5c-kserve-5777c77469-hcrrr\" (UID: \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\") " pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" Oct 30 19:26:29.188360 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.188330 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-tls-certs\") pod \"autores-model-c60f5c-kserve-5777c77469-hcrrr\" (UID: \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\") " pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" Oct 30 19:26:29.232112 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.231944 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-c60f5c\"/\"default-dockercfg-5xnjl\"" Oct 30 19:26:29.232112 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.232063 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-c60f5c\"/\"autores-model-c60f5c-kserve-self-signed-certs\"" Oct 30 19:26:29.239308 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.239271 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr"] Oct 30 19:26:29.289439 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.289391 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-tmp-dir\") pod \"autores-model-c60f5c-kserve-5777c77469-hcrrr\" (UID: \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\") " pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" Oct 30 19:26:29.289670 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.289477 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-tls-certs\") pod \"autores-model-c60f5c-kserve-5777c77469-hcrrr\" (UID: \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\") " pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" Oct 30 19:26:29.289670 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.289556 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-dshm\") pod \"autores-model-c60f5c-kserve-5777c77469-hcrrr\" (UID: \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\") " pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" Oct 30 19:26:29.289670 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.289582 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-model-cache\") pod \"autores-model-c60f5c-kserve-5777c77469-hcrrr\" (UID: \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\") " pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" Oct 30 19:26:29.289670 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.289620 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-home\") pod \"autores-model-c60f5c-kserve-5777c77469-hcrrr\" (UID: \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\") " pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" Oct 30 19:26:29.289915 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.289693 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f787f\" (UniqueName: \"kubernetes.io/projected/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-kube-api-access-f787f\") pod \"autores-model-c60f5c-kserve-5777c77469-hcrrr\" (UID: \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\") " pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" Oct 30 19:26:29.289915 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.289890 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-tmp-dir\") pod \"autores-model-c60f5c-kserve-5777c77469-hcrrr\" (UID: \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\") " pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" Oct 30 19:26:29.290091 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.289949 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-model-cache\") pod \"autores-model-c60f5c-kserve-5777c77469-hcrrr\" (UID: \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\") " pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" Oct 30 19:26:29.290191 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.290163 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-home\") pod \"autores-model-c60f5c-kserve-5777c77469-hcrrr\" (UID: \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\") " pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" Oct 30 19:26:29.292081 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.292051 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-dshm\") pod \"autores-model-c60f5c-kserve-5777c77469-hcrrr\" (UID: \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\") " pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" Oct 30 19:26:29.292606 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.292588 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-tls-certs\") pod \"autores-model-c60f5c-kserve-5777c77469-hcrrr\" (UID: \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\") " pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" Oct 30 19:26:29.352000 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.351954 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f787f\" (UniqueName: \"kubernetes.io/projected/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-kube-api-access-f787f\") pod \"autores-model-c60f5c-kserve-5777c77469-hcrrr\" (UID: \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\") " pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" Oct 30 19:26:29.435374 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.435266 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" Oct 30 19:26:29.824843 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:29.824815 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr"] Oct 30 19:26:29.826560 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:26:29.826534 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5ddc4c6_76c5_4040_8ed3_ec4161ad4b2e.slice/crio-57d8d611147544c6c95bc0fcf11af21274f6978b415bb970f4884f9fd986ff80 WatchSource:0}: Error finding container 57d8d611147544c6c95bc0fcf11af21274f6978b415bb970f4884f9fd986ff80: Status 404 returned error can't find the container with id 57d8d611147544c6c95bc0fcf11af21274f6978b415bb970f4884f9fd986ff80 Oct 30 19:26:30.183375 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:30.183288 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" event={"ID":"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e","Type":"ContainerStarted","Data":"dafab298d04711f03d27cf700b6dd22d036b3ae3cd21de7eadde9c4a0be18447"} Oct 30 19:26:30.183375 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:30.183334 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" event={"ID":"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e","Type":"ContainerStarted","Data":"57d8d611147544c6c95bc0fcf11af21274f6978b415bb970f4884f9fd986ff80"} Oct 30 19:26:30.303699 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:30.303645 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" podStartSLOduration=1.30362824 podStartE2EDuration="1.30362824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:30.300714904 +0000 UTC m=+3150.327819866" watchObservedRunningTime="2026-10-30 19:26:30.30362824 +0000 UTC m=+3150.330733198" Oct 30 19:26:30.435753 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:30.435658 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" Oct 30 19:26:38.867234 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:38.867197 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5"] Oct 30 19:26:38.870900 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:38.870879 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5" Oct 30 19:26:38.923244 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:38.923214 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:26:38.926000 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:38.925971 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:26:38.973153 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:38.973072 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:26:38.973363 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:38.973165 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:26:38.978147 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:38.978075 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5"] Oct 30 19:26:39.003885 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:39.003844 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f87e49e-4c36-4df8-9c37-42537b5dc289-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5\" (UID: \"3f87e49e-4c36-4df8-9c37-42537b5dc289\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5" Oct 30 19:26:39.004095 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:39.003897 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f87e49e-4c36-4df8-9c37-42537b5dc289-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5\" (UID: \"3f87e49e-4c36-4df8-9c37-42537b5dc289\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5" Oct 30 19:26:39.004095 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:39.004078 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb7c9\" (UniqueName: \"kubernetes.io/projected/3f87e49e-4c36-4df8-9c37-42537b5dc289-kube-api-access-jb7c9\") pod \"maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5\" (UID: \"3f87e49e-4c36-4df8-9c37-42537b5dc289\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5" Oct 30 19:26:39.105671 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:39.105623 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f87e49e-4c36-4df8-9c37-42537b5dc289-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5\" (UID: \"3f87e49e-4c36-4df8-9c37-42537b5dc289\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5" Oct 30 19:26:39.105671 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:39.105673 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f87e49e-4c36-4df8-9c37-42537b5dc289-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5\" (UID: \"3f87e49e-4c36-4df8-9c37-42537b5dc289\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5" Oct 30 19:26:39.105904 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:39.105735 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb7c9\" (UniqueName: \"kubernetes.io/projected/3f87e49e-4c36-4df8-9c37-42537b5dc289-kube-api-access-jb7c9\") pod \"maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5\" (UID: \"3f87e49e-4c36-4df8-9c37-42537b5dc289\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5" Oct 30 19:26:39.106516 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:39.106485 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f87e49e-4c36-4df8-9c37-42537b5dc289-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5\" (UID: \"3f87e49e-4c36-4df8-9c37-42537b5dc289\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5" Oct 30 19:26:39.108359 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:39.108332 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f87e49e-4c36-4df8-9c37-42537b5dc289-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5\" (UID: \"3f87e49e-4c36-4df8-9c37-42537b5dc289\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5" Oct 30 19:26:39.167555 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:39.167482 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb7c9\" (UniqueName: \"kubernetes.io/projected/3f87e49e-4c36-4df8-9c37-42537b5dc289-kube-api-access-jb7c9\") pod \"maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5\" (UID: \"3f87e49e-4c36-4df8-9c37-42537b5dc289\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5" Oct 30 19:26:39.185222 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:39.185190 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5" Oct 30 19:26:39.435465 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:39.435369 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" Oct 30 19:26:39.449651 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:39.449618 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" Oct 30 19:26:39.567046 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:39.566999 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5"] Oct 30 19:26:39.569691 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:26:39.569657 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f87e49e_4c36_4df8_9c37_42537b5dc289.slice/crio-bdb31ac6af4ac4a50cff1879e0b72952bdb9c3abb4e3e1805ae83a609999a2f8 WatchSource:0}: Error finding container bdb31ac6af4ac4a50cff1879e0b72952bdb9c3abb4e3e1805ae83a609999a2f8: Status 404 returned error can't find the container with id bdb31ac6af4ac4a50cff1879e0b72952bdb9c3abb4e3e1805ae83a609999a2f8 Oct 30 19:26:40.242886 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:40.242854 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5" event={"ID":"3f87e49e-4c36-4df8-9c37-42537b5dc289","Type":"ContainerStarted","Data":"d26bd5cf91f3d730e5adb28d86464ba053a8e25b3531f6a8c97908de237714e3"} Oct 30 19:26:40.243392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:40.242910 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5" event={"ID":"3f87e49e-4c36-4df8-9c37-42537b5dc289","Type":"ContainerStarted","Data":"bdb31ac6af4ac4a50cff1879e0b72952bdb9c3abb4e3e1805ae83a609999a2f8"} Oct 30 19:26:40.254394 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:40.254353 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" Oct 30 19:26:40.363812 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:40.363753 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5" podStartSLOduration=2.363734823 podStartE2EDuration="2.363734823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:40.361727246 +0000 UTC m=+3160.388832216" watchObservedRunningTime="2026-10-30 19:26:40.363734823 +0000 UTC m=+3160.390839782" Oct 30 19:26:41.251259 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:41.251219 4531 generic.go:356] "Generic (PLEG): container finished" podID="3f87e49e-4c36-4df8-9c37-42537b5dc289" containerID="d26bd5cf91f3d730e5adb28d86464ba053a8e25b3531f6a8c97908de237714e3" exitCode=0 Oct 30 19:26:41.251762 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:41.251297 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5" event={"ID":"3f87e49e-4c36-4df8-9c37-42537b5dc289","Type":"ContainerDied","Data":"d26bd5cf91f3d730e5adb28d86464ba053a8e25b3531f6a8c97908de237714e3"} Oct 30 19:26:42.411175 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:42.411120 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5" Oct 30 19:26:42.446550 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:42.446508 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f87e49e-4c36-4df8-9c37-42537b5dc289-kube-api-access-jb7c9\" (UniqueName: \"kubernetes.io/projected/3f87e49e-4c36-4df8-9c37-42537b5dc289-kube-api-access-jb7c9\") pod \"3f87e49e-4c36-4df8-9c37-42537b5dc289\" (UID: \"3f87e49e-4c36-4df8-9c37-42537b5dc289\") " Oct 30 19:26:42.446735 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:42.446627 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f87e49e-4c36-4df8-9c37-42537b5dc289-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f87e49e-4c36-4df8-9c37-42537b5dc289-maas-api-xks-ca\") pod \"3f87e49e-4c36-4df8-9c37-42537b5dc289\" (UID: \"3f87e49e-4c36-4df8-9c37-42537b5dc289\") " Oct 30 19:26:42.446735 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:42.446655 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f87e49e-4c36-4df8-9c37-42537b5dc289-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f87e49e-4c36-4df8-9c37-42537b5dc289-maas-api-openshift-ca\") pod \"3f87e49e-4c36-4df8-9c37-42537b5dc289\" (UID: \"3f87e49e-4c36-4df8-9c37-42537b5dc289\") " Oct 30 19:26:42.447090 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:42.447055 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f87e49e-4c36-4df8-9c37-42537b5dc289-maas-api-openshift-ca" pod "3f87e49e-4c36-4df8-9c37-42537b5dc289" (UID: "3f87e49e-4c36-4df8-9c37-42537b5dc289"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:26:42.448997 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:42.448969 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f87e49e-4c36-4df8-9c37-42537b5dc289-kube-api-access-jb7c9" pod "3f87e49e-4c36-4df8-9c37-42537b5dc289" (UID: "3f87e49e-4c36-4df8-9c37-42537b5dc289"). InnerVolumeSpecName "kube-api-access-jb7c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:26:42.449146 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:42.448995 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f87e49e-4c36-4df8-9c37-42537b5dc289-maas-api-xks-ca" pod "3f87e49e-4c36-4df8-9c37-42537b5dc289" (UID: "3f87e49e-4c36-4df8-9c37-42537b5dc289"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:26:42.548210 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:42.548081 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f87e49e-4c36-4df8-9c37-42537b5dc289-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:26:42.548210 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:42.548121 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f87e49e-4c36-4df8-9c37-42537b5dc289-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:26:42.548210 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:42.548179 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb7c9\" (UniqueName: \"kubernetes.io/projected/3f87e49e-4c36-4df8-9c37-42537b5dc289-kube-api-access-jb7c9\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:26:43.265610 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:43.265580 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5" Oct 30 19:26:43.265783 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:43.265611 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5" event={"ID":"3f87e49e-4c36-4df8-9c37-42537b5dc289","Type":"ContainerDied","Data":"bdb31ac6af4ac4a50cff1879e0b72952bdb9c3abb4e3e1805ae83a609999a2f8"} Oct 30 19:26:43.265783 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:26:43.265664 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdb31ac6af4ac4a50cff1879e0b72952bdb9c3abb4e3e1805ae83a609999a2f8" Oct 30 19:26:56.400397 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:26:56.400351 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:27:17.006299 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:17.006207 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr"] Oct 30 19:27:17.604710 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:17.604653 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h"] Oct 30 19:27:17.605188 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:17.605172 4531 cpu_manager.go:414] "containerMap: removing container" podUID="3f87e49e-4c36-4df8-9c37-42537b5dc289" containerName="revoke-keys" Oct 30 19:27:17.605188 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:17.605188 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f87e49e-4c36-4df8-9c37-42537b5dc289" containerName="revoke-keys" Oct 30 19:27:17.605331 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:17.605302 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f87e49e-4c36-4df8-9c37-42537b5dc289" containerName="revoke-keys" Oct 30 19:27:17.609926 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:17.609892 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h" Oct 30 19:27:17.665206 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:17.665167 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:27:17.673310 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:17.673272 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:27:17.684007 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:17.683962 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h"] Oct 30 19:27:17.706484 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:17.706434 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfe0f0c8-8f23-4828-b2c7-c46f92d158fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h\" (UID: \"bfe0f0c8-8f23-4828-b2c7-c46f92d158fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h" Oct 30 19:27:17.706708 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:17.706582 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6fmw\" (UniqueName: \"kubernetes.io/projected/bfe0f0c8-8f23-4828-b2c7-c46f92d158fc-kube-api-access-q6fmw\") pod \"maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h\" (UID: \"bfe0f0c8-8f23-4828-b2c7-c46f92d158fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h" Oct 30 19:27:17.706708 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:17.706634 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfe0f0c8-8f23-4828-b2c7-c46f92d158fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h\" (UID: \"bfe0f0c8-8f23-4828-b2c7-c46f92d158fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h" Oct 30 19:27:17.714529 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:17.714481 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:27:17.714529 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:17.714519 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:27:17.807476 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:17.807417 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6fmw\" (UniqueName: \"kubernetes.io/projected/bfe0f0c8-8f23-4828-b2c7-c46f92d158fc-kube-api-access-q6fmw\") pod \"maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h\" (UID: \"bfe0f0c8-8f23-4828-b2c7-c46f92d158fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h" Oct 30 19:27:17.807476 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:17.807483 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfe0f0c8-8f23-4828-b2c7-c46f92d158fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h\" (UID: \"bfe0f0c8-8f23-4828-b2c7-c46f92d158fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h" Oct 30 19:27:17.807758 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:17.807712 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfe0f0c8-8f23-4828-b2c7-c46f92d158fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h\" (UID: \"bfe0f0c8-8f23-4828-b2c7-c46f92d158fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h" Oct 30 19:27:17.808586 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:17.808556 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfe0f0c8-8f23-4828-b2c7-c46f92d158fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h\" (UID: \"bfe0f0c8-8f23-4828-b2c7-c46f92d158fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h" Oct 30 19:27:17.810301 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:17.810266 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfe0f0c8-8f23-4828-b2c7-c46f92d158fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h\" (UID: \"bfe0f0c8-8f23-4828-b2c7-c46f92d158fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h" Oct 30 19:27:17.869494 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:17.869403 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6fmw\" (UniqueName: \"kubernetes.io/projected/bfe0f0c8-8f23-4828-b2c7-c46f92d158fc-kube-api-access-q6fmw\") pod \"maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h\" (UID: \"bfe0f0c8-8f23-4828-b2c7-c46f92d158fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h" Oct 30 19:27:17.926824 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:17.926792 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h" Oct 30 19:27:18.313080 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:18.313043 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h"] Oct 30 19:27:18.314770 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:27:18.314734 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe0f0c8_8f23_4828_b2c7_c46f92d158fc.slice/crio-339cbd71e3b7b8950597c5cae802217676e1c0bd374dea13dbaae71eb640a161 WatchSource:0}: Error finding container 339cbd71e3b7b8950597c5cae802217676e1c0bd374dea13dbaae71eb640a161: Status 404 returned error can't find the container with id 339cbd71e3b7b8950597c5cae802217676e1c0bd374dea13dbaae71eb640a161 Oct 30 19:27:18.477871 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:18.477826 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h" event={"ID":"bfe0f0c8-8f23-4828-b2c7-c46f92d158fc","Type":"ContainerStarted","Data":"339cbd71e3b7b8950597c5cae802217676e1c0bd374dea13dbaae71eb640a161"} Oct 30 19:27:19.485759 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:19.485718 4531 generic.go:356] "Generic (PLEG): container finished" podID="bfe0f0c8-8f23-4828-b2c7-c46f92d158fc" containerID="1576a4e547ae1cf82e5adf180c6bc939c7d4e895b182aca0a98fd6978ebb8d33" exitCode=0 Oct 30 19:27:19.486247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:19.485810 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h" event={"ID":"bfe0f0c8-8f23-4828-b2c7-c46f92d158fc","Type":"ContainerDied","Data":"1576a4e547ae1cf82e5adf180c6bc939c7d4e895b182aca0a98fd6978ebb8d33"} Oct 30 19:27:20.646662 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:20.646636 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h" Oct 30 19:27:20.744821 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:20.744724 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe0f0c8-8f23-4828-b2c7-c46f92d158fc-kube-api-access-q6fmw\" (UniqueName: \"kubernetes.io/projected/bfe0f0c8-8f23-4828-b2c7-c46f92d158fc-kube-api-access-q6fmw\") pod \"bfe0f0c8-8f23-4828-b2c7-c46f92d158fc\" (UID: \"bfe0f0c8-8f23-4828-b2c7-c46f92d158fc\") " Oct 30 19:27:20.745027 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:20.744834 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfe0f0c8-8f23-4828-b2c7-c46f92d158fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfe0f0c8-8f23-4828-b2c7-c46f92d158fc-maas-api-openshift-ca\") pod \"bfe0f0c8-8f23-4828-b2c7-c46f92d158fc\" (UID: \"bfe0f0c8-8f23-4828-b2c7-c46f92d158fc\") " Oct 30 19:27:20.745027 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:20.745021 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfe0f0c8-8f23-4828-b2c7-c46f92d158fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfe0f0c8-8f23-4828-b2c7-c46f92d158fc-maas-api-xks-ca\") pod \"bfe0f0c8-8f23-4828-b2c7-c46f92d158fc\" (UID: \"bfe0f0c8-8f23-4828-b2c7-c46f92d158fc\") " Oct 30 19:27:20.745276 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:20.745251 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe0f0c8-8f23-4828-b2c7-c46f92d158fc-maas-api-openshift-ca" pod "bfe0f0c8-8f23-4828-b2c7-c46f92d158fc" (UID: "bfe0f0c8-8f23-4828-b2c7-c46f92d158fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:27:20.745526 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:20.745508 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfe0f0c8-8f23-4828-b2c7-c46f92d158fc-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:27:20.747240 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:20.747210 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfe0f0c8-8f23-4828-b2c7-c46f92d158fc-maas-api-xks-ca" pod "bfe0f0c8-8f23-4828-b2c7-c46f92d158fc" (UID: "bfe0f0c8-8f23-4828-b2c7-c46f92d158fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:27:20.747356 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:20.747244 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe0f0c8-8f23-4828-b2c7-c46f92d158fc-kube-api-access-q6fmw" pod "bfe0f0c8-8f23-4828-b2c7-c46f92d158fc" (UID: "bfe0f0c8-8f23-4828-b2c7-c46f92d158fc"). InnerVolumeSpecName "kube-api-access-q6fmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:20.846892 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:20.846851 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6fmw\" (UniqueName: \"kubernetes.io/projected/bfe0f0c8-8f23-4828-b2c7-c46f92d158fc-kube-api-access-q6fmw\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:27:20.846892 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:20.846886 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfe0f0c8-8f23-4828-b2c7-c46f92d158fc-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:27:21.499456 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:21.499420 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h" Oct 30 19:27:21.499456 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:21.499449 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h" event={"ID":"bfe0f0c8-8f23-4828-b2c7-c46f92d158fc","Type":"ContainerDied","Data":"339cbd71e3b7b8950597c5cae802217676e1c0bd374dea13dbaae71eb640a161"} Oct 30 19:27:21.499742 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:21.499497 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="339cbd71e3b7b8950597c5cae802217676e1c0bd374dea13dbaae71eb640a161" Oct 30 19:27:26.398663 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:27:26.398611 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:27:27.085157 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:27.085085 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbb380da-487ts"] Oct 30 19:27:27.135110 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:27.135072 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbb380da-487ts"] Oct 30 19:27:28.098666 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:28.098632 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33307e75-c8a6-45a1-b6ab-959ffd695d99" path="/var/lib/kubelet/pods/33307e75-c8a6-45a1-b6ab-959ffd695d99/volumes" Oct 30 19:27:32.027901 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.027852 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" podUID="a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e" containerName="main" containerID="cri-o://dafab298d04711f03d27cf700b6dd22d036b3ae3cd21de7eadde9c4a0be18447" gracePeriod=45 Oct 30 19:27:32.304949 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.304912 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" Oct 30 19:27:32.386819 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.386779 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-home\" (UniqueName: \"kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-home\") pod \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\" (UID: \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\") " Oct 30 19:27:32.387048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.386851 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-tmp-dir\") pod \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\" (UID: \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\") " Oct 30 19:27:32.387048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.386885 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-model-cache\") pod \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\" (UID: \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\") " Oct 30 19:27:32.387048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.386948 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-kube-api-access-f787f\" (UniqueName: \"kubernetes.io/projected/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-kube-api-access-f787f\") pod \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\" (UID: \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\") " Oct 30 19:27:32.387048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.387005 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-tls-certs\") pod \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\" (UID: \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\") " Oct 30 19:27:32.387378 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.387123 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-tmp-dir" pod "a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e" (UID: "a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:27:32.387378 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.387182 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-model-cache" pod "a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e" (UID: "a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:27:32.387378 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.387122 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-home" pod "a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e" (UID: "a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:27:32.387378 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.387188 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-dshm\") pod \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\" (UID: \"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e\") " Oct 30 19:27:32.387859 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.387671 4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-model-cache\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:27:32.387941 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.387864 4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-home\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:27:32.387941 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.387879 4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-tmp-dir\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:27:32.389564 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.389531 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-tls-certs" pod "a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e" (UID: "a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:27:32.389564 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.389556 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-dshm" pod "a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e" (UID: "a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:27:32.389720 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.389598 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-kube-api-access-f787f" pod "a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e" (UID: "a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e"). InnerVolumeSpecName "kube-api-access-f787f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:32.488686 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.488646 4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-dshm\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:27:32.488686 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.488681 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f787f\" (UniqueName: \"kubernetes.io/projected/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-kube-api-access-f787f\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:27:32.488686 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.488694 4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e-tls-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:27:32.567598 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.567514 4531 generic.go:356] "Generic (PLEG): container finished" podID="a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e" containerID="dafab298d04711f03d27cf700b6dd22d036b3ae3cd21de7eadde9c4a0be18447" exitCode=0 Oct 30 19:27:32.567598 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.567560 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" Oct 30 19:27:32.567598 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.567588 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" event={"ID":"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e","Type":"ContainerDied","Data":"dafab298d04711f03d27cf700b6dd22d036b3ae3cd21de7eadde9c4a0be18447"} Oct 30 19:27:32.567849 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.567633 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" event={"ID":"a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e","Type":"ContainerDied","Data":"57d8d611147544c6c95bc0fcf11af21274f6978b415bb970f4884f9fd986ff80"} Oct 30 19:27:32.567849 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.567653 4531 scope.go:122] "RemoveContainer" containerID="dafab298d04711f03d27cf700b6dd22d036b3ae3cd21de7eadde9c4a0be18447" Oct 30 19:27:32.577798 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.577771 4531 scope.go:122] "RemoveContainer" containerID="dafab298d04711f03d27cf700b6dd22d036b3ae3cd21de7eadde9c4a0be18447" Oct 30 19:27:32.578197 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:27:32.578146 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dafab298d04711f03d27cf700b6dd22d036b3ae3cd21de7eadde9c4a0be18447\": container with ID starting with dafab298d04711f03d27cf700b6dd22d036b3ae3cd21de7eadde9c4a0be18447 not found: ID does not exist" containerID="dafab298d04711f03d27cf700b6dd22d036b3ae3cd21de7eadde9c4a0be18447" Oct 30 19:27:32.578197 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.578178 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dafab298d04711f03d27cf700b6dd22d036b3ae3cd21de7eadde9c4a0be18447"} err="failed to get container status \"dafab298d04711f03d27cf700b6dd22d036b3ae3cd21de7eadde9c4a0be18447\": rpc error: code = NotFound desc = could not find container \"dafab298d04711f03d27cf700b6dd22d036b3ae3cd21de7eadde9c4a0be18447\": container with ID starting with dafab298d04711f03d27cf700b6dd22d036b3ae3cd21de7eadde9c4a0be18447 not found: ID does not exist" Oct 30 19:27:32.734592 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.734548 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr"] Oct 30 19:27:32.738587 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:32.738552 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr"] Oct 30 19:27:33.244219 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:33.244113 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-c60f5c/autores-model-c60f5c-kserve-5777c77469-hcrrr" podUID="a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.224:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:34.099430 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:34.099380 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e" path="/var/lib/kubelet/pods/a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e/volumes" Oct 30 19:27:34.603063 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:34.603030 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h"] Oct 30 19:27:34.653689 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:34.653645 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c60f5c-qxr6h"] Oct 30 19:27:36.100755 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:36.100707 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfe0f0c8-8f23-4828-b2c7-c46f92d158fc" path="/var/lib/kubelet/pods/bfe0f0c8-8f23-4828-b2c7-c46f92d158fc/volumes" Oct 30 19:27:42.100523 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:42.100490 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc43760f-sdcj7"] Oct 30 19:27:42.100937 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:42.100910 4531 cpu_manager.go:414] "containerMap: removing container" podUID="a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e" containerName="main" Oct 30 19:27:42.100937 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:42.100922 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e" containerName="main" Oct 30 19:27:42.101017 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:42.100960 4531 cpu_manager.go:414] "containerMap: removing container" podUID="bfe0f0c8-8f23-4828-b2c7-c46f92d158fc" containerName="revoke-keys" Oct 30 19:27:42.101017 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:42.100965 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfe0f0c8-8f23-4828-b2c7-c46f92d158fc" containerName="revoke-keys" Oct 30 19:27:42.101083 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:42.101052 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfe0f0c8-8f23-4828-b2c7-c46f92d158fc" containerName="revoke-keys" Oct 30 19:27:42.101083 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:42.101063 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5ddc4c6-76c5-4040-8ed3-ec4161ad4b2e" containerName="main" Oct 30 19:27:42.105350 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:42.105324 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc43760f-sdcj7" Oct 30 19:27:42.162815 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:42.162770 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc43760f-sdcj7"] Oct 30 19:27:42.202010 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:42.201974 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:27:42.202229 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:42.201978 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:27:42.204644 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:42.204611 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf335c92-b015-4e59-9142-181bd2e7aee1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc43760f-sdcj7\" (UID: \"cf335c92-b015-4e59-9142-181bd2e7aee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc43760f-sdcj7" Oct 30 19:27:42.204827 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:42.204665 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf335c92-b015-4e59-9142-181bd2e7aee1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc43760f-sdcj7\" (UID: \"cf335c92-b015-4e59-9142-181bd2e7aee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc43760f-sdcj7" Oct 30 19:27:42.204827 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:42.204764 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ncrg\" (UniqueName: \"kubernetes.io/projected/cf335c92-b015-4e59-9142-181bd2e7aee1-kube-api-access-6ncrg\") pod \"maas-api-revoke-sub-fc43760f-sdcj7\" (UID: \"cf335c92-b015-4e59-9142-181bd2e7aee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc43760f-sdcj7" Oct 30 19:27:42.208449 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:42.208423 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:27:42.214334 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:42.214306 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:27:42.306528 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:42.306483 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf335c92-b015-4e59-9142-181bd2e7aee1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc43760f-sdcj7\" (UID: \"cf335c92-b015-4e59-9142-181bd2e7aee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc43760f-sdcj7" Oct 30 19:27:42.306772 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:42.306552 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf335c92-b015-4e59-9142-181bd2e7aee1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc43760f-sdcj7\" (UID: \"cf335c92-b015-4e59-9142-181bd2e7aee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc43760f-sdcj7" Oct 30 19:27:42.306772 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:42.306574 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ncrg\" (UniqueName: \"kubernetes.io/projected/cf335c92-b015-4e59-9142-181bd2e7aee1-kube-api-access-6ncrg\") pod \"maas-api-revoke-sub-fc43760f-sdcj7\" (UID: \"cf335c92-b015-4e59-9142-181bd2e7aee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc43760f-sdcj7" Oct 30 19:27:42.307320 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:42.307293 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf335c92-b015-4e59-9142-181bd2e7aee1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc43760f-sdcj7\" (UID: \"cf335c92-b015-4e59-9142-181bd2e7aee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc43760f-sdcj7" Oct 30 19:27:42.309246 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:42.309222 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf335c92-b015-4e59-9142-181bd2e7aee1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc43760f-sdcj7\" (UID: \"cf335c92-b015-4e59-9142-181bd2e7aee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc43760f-sdcj7" Oct 30 19:27:42.368213 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:42.368109 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ncrg\" (UniqueName: \"kubernetes.io/projected/cf335c92-b015-4e59-9142-181bd2e7aee1-kube-api-access-6ncrg\") pod \"maas-api-revoke-sub-fc43760f-sdcj7\" (UID: \"cf335c92-b015-4e59-9142-181bd2e7aee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc43760f-sdcj7" Oct 30 19:27:42.418460 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:42.418421 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc43760f-sdcj7" Oct 30 19:27:42.800578 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:42.800548 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc43760f-sdcj7"] Oct 30 19:27:42.803012 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:27:42.802979 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf335c92_b015_4e59_9142_181bd2e7aee1.slice/crio-9c555e8bb719e012a1a20b50cbb3be7c209b2a67de8e323a7e792f1de72e7070 WatchSource:0}: Error finding container 9c555e8bb719e012a1a20b50cbb3be7c209b2a67de8e323a7e792f1de72e7070: Status 404 returned error can't find the container with id 9c555e8bb719e012a1a20b50cbb3be7c209b2a67de8e323a7e792f1de72e7070 Oct 30 19:27:43.637571 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:43.637525 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc43760f-sdcj7" event={"ID":"cf335c92-b015-4e59-9142-181bd2e7aee1","Type":"ContainerStarted","Data":"e82d8e853cde3d8a69668f7ddaff6fd2bc5b0d99ab2320ffff549b0e9b655f30"} Oct 30 19:27:43.638110 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:43.637635 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc43760f-sdcj7" event={"ID":"cf335c92-b015-4e59-9142-181bd2e7aee1","Type":"ContainerStarted","Data":"9c555e8bb719e012a1a20b50cbb3be7c209b2a67de8e323a7e792f1de72e7070"} Oct 30 19:27:43.757609 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:43.757548 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc43760f-sdcj7" podStartSLOduration=1.75753182 podStartE2EDuration="1.75753182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:43.755751516 +0000 UTC m=+3223.782856476" watchObservedRunningTime="2026-10-30 19:27:43.75753182 +0000 UTC m=+3223.784636780" Oct 30 19:27:44.645496 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:44.645450 4531 generic.go:356] "Generic (PLEG): container finished" podID="cf335c92-b015-4e59-9142-181bd2e7aee1" containerID="e82d8e853cde3d8a69668f7ddaff6fd2bc5b0d99ab2320ffff549b0e9b655f30" exitCode=0 Oct 30 19:27:44.645922 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:44.645536 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc43760f-sdcj7" event={"ID":"cf335c92-b015-4e59-9142-181bd2e7aee1","Type":"ContainerDied","Data":"e82d8e853cde3d8a69668f7ddaff6fd2bc5b0d99ab2320ffff549b0e9b655f30"} Oct 30 19:27:45.811511 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:45.811484 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc43760f-sdcj7" Oct 30 19:27:45.954682 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:45.954588 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf335c92-b015-4e59-9142-181bd2e7aee1-kube-api-access-6ncrg\" (UniqueName: \"kubernetes.io/projected/cf335c92-b015-4e59-9142-181bd2e7aee1-kube-api-access-6ncrg\") pod \"cf335c92-b015-4e59-9142-181bd2e7aee1\" (UID: \"cf335c92-b015-4e59-9142-181bd2e7aee1\") " Oct 30 19:27:45.954856 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:45.954696 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf335c92-b015-4e59-9142-181bd2e7aee1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf335c92-b015-4e59-9142-181bd2e7aee1-maas-api-openshift-ca\") pod \"cf335c92-b015-4e59-9142-181bd2e7aee1\" (UID: \"cf335c92-b015-4e59-9142-181bd2e7aee1\") " Oct 30 19:27:45.954856 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:45.954776 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf335c92-b015-4e59-9142-181bd2e7aee1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf335c92-b015-4e59-9142-181bd2e7aee1-maas-api-xks-ca\") pod \"cf335c92-b015-4e59-9142-181bd2e7aee1\" (UID: \"cf335c92-b015-4e59-9142-181bd2e7aee1\") " Oct 30 19:27:45.957117 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:45.957082 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf335c92-b015-4e59-9142-181bd2e7aee1-kube-api-access-6ncrg" pod "cf335c92-b015-4e59-9142-181bd2e7aee1" (UID: "cf335c92-b015-4e59-9142-181bd2e7aee1"). InnerVolumeSpecName "kube-api-access-6ncrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:45.957117 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:45.957099 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf335c92-b015-4e59-9142-181bd2e7aee1-maas-api-xks-ca" pod "cf335c92-b015-4e59-9142-181bd2e7aee1" (UID: "cf335c92-b015-4e59-9142-181bd2e7aee1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:27:45.969185 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:45.969110 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf335c92-b015-4e59-9142-181bd2e7aee1-maas-api-openshift-ca" pod "cf335c92-b015-4e59-9142-181bd2e7aee1" (UID: "cf335c92-b015-4e59-9142-181bd2e7aee1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:27:46.056484 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:46.056451 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ncrg\" (UniqueName: \"kubernetes.io/projected/cf335c92-b015-4e59-9142-181bd2e7aee1-kube-api-access-6ncrg\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:27:46.056484 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:46.056478 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf335c92-b015-4e59-9142-181bd2e7aee1-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:27:46.056484 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:46.056490 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf335c92-b015-4e59-9142-181bd2e7aee1-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:27:46.659053 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:46.659019 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc43760f-sdcj7" Oct 30 19:27:46.659294 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:46.659053 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc43760f-sdcj7" event={"ID":"cf335c92-b015-4e59-9142-181bd2e7aee1","Type":"ContainerDied","Data":"9c555e8bb719e012a1a20b50cbb3be7c209b2a67de8e323a7e792f1de72e7070"} Oct 30 19:27:46.659294 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:27:46.659102 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c555e8bb719e012a1a20b50cbb3be7c209b2a67de8e323a7e792f1de72e7070" Oct 30 19:27:56.402356 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:27:56.402314 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:28:00.757976 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:00.757932 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48"] Oct 30 19:28:00.758677 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:00.758253 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" podUID="b5ecb161-fdf5-4084-9080-0227a442d88a" containerName="istio-proxy" containerID="cri-o://e095ac284e2cafb5255bfacf6d35d9c2f4431284b0b3d5d1d2dbbc5dcaf5daf2" gracePeriod=30 Oct 30 19:28:02.714007 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:02.713973 4531 scope.go:122] "RemoveContainer" containerID="4fcad77456f3b8b4ee0dfc146497f7db934027aca0fb9b572d0e11b581893140" Oct 30 19:28:02.724689 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:02.724663 4531 scope.go:122] "RemoveContainer" containerID="0fdd4bfa6a8356466d68a6cfd73d885ca6fed9622b9a32740f6f9280332e63c9" Oct 30 19:28:04.274261 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:04.274222 4531 patch_prober.go:29] interesting pod/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:28:04.274673 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:04.274277 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" podUID="b5ecb161-fdf5-4084-9080-0227a442d88a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:28:06.026234 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.026167 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:28:06.091757 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.091713 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5ecb161-fdf5-4084-9080-0227a442d88a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ecb161-fdf5-4084-9080-0227a442d88a-istiod-ca-cert\") pod \"b5ecb161-fdf5-4084-9080-0227a442d88a\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " Oct 30 19:28:06.091963 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.091775 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-data\") pod \"b5ecb161-fdf5-4084-9080-0227a442d88a\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " Oct 30 19:28:06.091963 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.091811 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5ecb161-fdf5-4084-9080-0227a442d88a-kube-api-access-c559f\" (UniqueName: \"kubernetes.io/projected/b5ecb161-fdf5-4084-9080-0227a442d88a-kube-api-access-c559f\") pod \"b5ecb161-fdf5-4084-9080-0227a442d88a\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " Oct 30 19:28:06.092095 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.091993 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-workload-socket\") pod \"b5ecb161-fdf5-4084-9080-0227a442d88a\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " Oct 30 19:28:06.092192 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.092113 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-data" pod "b5ecb161-fdf5-4084-9080-0227a442d88a" (UID: "b5ecb161-fdf5-4084-9080-0227a442d88a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:06.092245 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.092195 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5ecb161-fdf5-4084-9080-0227a442d88a-istiod-ca-cert" pod "b5ecb161-fdf5-4084-9080-0227a442d88a" (UID: "b5ecb161-fdf5-4084-9080-0227a442d88a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:28:06.092358 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.092329 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-workload-socket" pod "b5ecb161-fdf5-4084-9080-0227a442d88a" (UID: "b5ecb161-fdf5-4084-9080-0227a442d88a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:06.092452 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.092364 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-token\" (UniqueName: \"kubernetes.io/projected/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-token\") pod \"b5ecb161-fdf5-4084-9080-0227a442d88a\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " Oct 30 19:28:06.092452 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.092410 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-credential-socket\") pod \"b5ecb161-fdf5-4084-9080-0227a442d88a\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " Oct 30 19:28:06.092452 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.092450 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-podinfo\") pod \"b5ecb161-fdf5-4084-9080-0227a442d88a\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " Oct 30 19:28:06.092627 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.092502 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-workload-certs\") pod \"b5ecb161-fdf5-4084-9080-0227a442d88a\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " Oct 30 19:28:06.092627 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.092547 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-envoy\") pod \"b5ecb161-fdf5-4084-9080-0227a442d88a\" (UID: \"b5ecb161-fdf5-4084-9080-0227a442d88a\") " Oct 30 19:28:06.092904 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.092679 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-credential-socket" pod "b5ecb161-fdf5-4084-9080-0227a442d88a" (UID: "b5ecb161-fdf5-4084-9080-0227a442d88a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:06.092904 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.092768 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-workload-certs" pod "b5ecb161-fdf5-4084-9080-0227a442d88a" (UID: "b5ecb161-fdf5-4084-9080-0227a442d88a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:06.093022 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.092962 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ecb161-fdf5-4084-9080-0227a442d88a-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:28:06.093022 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.092981 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:28:06.093022 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.092995 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:28:06.093022 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.093009 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:28:06.093305 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.093023 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:28:06.094555 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.094525 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5ecb161-fdf5-4084-9080-0227a442d88a-kube-api-access-c559f" pod "b5ecb161-fdf5-4084-9080-0227a442d88a" (UID: "b5ecb161-fdf5-4084-9080-0227a442d88a"). InnerVolumeSpecName "kube-api-access-c559f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:28:06.094971 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.094939 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-envoy" pod "b5ecb161-fdf5-4084-9080-0227a442d88a" (UID: "b5ecb161-fdf5-4084-9080-0227a442d88a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:06.095269 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.095247 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-token" pod "b5ecb161-fdf5-4084-9080-0227a442d88a" (UID: "b5ecb161-fdf5-4084-9080-0227a442d88a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:28:06.095357 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.095252 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-podinfo" pod "b5ecb161-fdf5-4084-9080-0227a442d88a" (UID: "b5ecb161-fdf5-4084-9080-0227a442d88a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:28:06.194866 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.194773 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:28:06.194866 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.194812 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:28:06.194866 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.194828 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ecb161-fdf5-4084-9080-0227a442d88a-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:28:06.194866 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.194844 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c559f\" (UniqueName: \"kubernetes.io/projected/b5ecb161-fdf5-4084-9080-0227a442d88a-kube-api-access-c559f\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:28:06.783400 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.783354 4531 generic.go:356] "Generic (PLEG): container finished" podID="b5ecb161-fdf5-4084-9080-0227a442d88a" containerID="e095ac284e2cafb5255bfacf6d35d9c2f4431284b0b3d5d1d2dbbc5dcaf5daf2" exitCode=0 Oct 30 19:28:06.783599 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.783408 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" Oct 30 19:28:06.783599 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.783436 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" event={"ID":"b5ecb161-fdf5-4084-9080-0227a442d88a","Type":"ContainerDied","Data":"e095ac284e2cafb5255bfacf6d35d9c2f4431284b0b3d5d1d2dbbc5dcaf5daf2"} Oct 30 19:28:06.783599 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.783503 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48" event={"ID":"b5ecb161-fdf5-4084-9080-0227a442d88a","Type":"ContainerDied","Data":"85e27ec20a9899838da673e3b6e7b09e0b7883b2a245d56b3eaa2df9aa62d4a8"} Oct 30 19:28:06.783599 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.783527 4531 scope.go:122] "RemoveContainer" containerID="e095ac284e2cafb5255bfacf6d35d9c2f4431284b0b3d5d1d2dbbc5dcaf5daf2" Oct 30 19:28:06.793469 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.793440 4531 scope.go:122] "RemoveContainer" containerID="e095ac284e2cafb5255bfacf6d35d9c2f4431284b0b3d5d1d2dbbc5dcaf5daf2" Oct 30 19:28:06.793853 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:28:06.793831 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e095ac284e2cafb5255bfacf6d35d9c2f4431284b0b3d5d1d2dbbc5dcaf5daf2\": container with ID starting with e095ac284e2cafb5255bfacf6d35d9c2f4431284b0b3d5d1d2dbbc5dcaf5daf2 not found: ID does not exist" containerID="e095ac284e2cafb5255bfacf6d35d9c2f4431284b0b3d5d1d2dbbc5dcaf5daf2" Oct 30 19:28:06.793938 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:06.793862 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e095ac284e2cafb5255bfacf6d35d9c2f4431284b0b3d5d1d2dbbc5dcaf5daf2"} err="failed to get container status \"e095ac284e2cafb5255bfacf6d35d9c2f4431284b0b3d5d1d2dbbc5dcaf5daf2\": rpc error: code = NotFound desc = could not find container \"e095ac284e2cafb5255bfacf6d35d9c2f4431284b0b3d5d1d2dbbc5dcaf5daf2\": container with ID starting with e095ac284e2cafb5255bfacf6d35d9c2f4431284b0b3d5d1d2dbbc5dcaf5daf2 not found: ID does not exist" Oct 30 19:28:07.000550 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:07.000511 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48"] Oct 30 19:28:07.050482 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:07.050386 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4a7a52-openshift-default-7cfbc88b69-d8c48"] Oct 30 19:28:08.099605 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:08.099568 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5ecb161-fdf5-4084-9080-0227a442d88a" path="/var/lib/kubelet/pods/b5ecb161-fdf5-4084-9080-0227a442d88a/volumes" Oct 30 19:28:14.297561 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:14.297520 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4"] Oct 30 19:28:14.298015 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:14.297746 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" podUID="660d18a9-a78d-49f8-8e10-537b4a257731" containerName="istio-proxy" containerID="cri-o://38c63315d96ea28e0551c2a21b38e602decdaf54bc2791faec85fc2e75d96b2d" gracePeriod=30 Oct 30 19:28:16.927142 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:16.927083 4531 patch_prober.go:29] interesting pod/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:28:16.927569 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:16.927165 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" podUID="660d18a9-a78d-49f8-8e10-537b4a257731" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:28:19.566845 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.566781 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:28:19.746047 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.746010 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660d18a9-a78d-49f8-8e10-537b4a257731-istio-token\" (UniqueName: \"kubernetes.io/projected/660d18a9-a78d-49f8-8e10-537b4a257731-istio-token\") pod \"660d18a9-a78d-49f8-8e10-537b4a257731\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " Oct 30 19:28:19.746244 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.746054 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-istio-data\") pod \"660d18a9-a78d-49f8-8e10-537b4a257731\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " Oct 30 19:28:19.746244 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.746074 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/660d18a9-a78d-49f8-8e10-537b4a257731-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660d18a9-a78d-49f8-8e10-537b4a257731-istio-podinfo\") pod \"660d18a9-a78d-49f8-8e10-537b4a257731\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " Oct 30 19:28:19.746244 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.746123 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-workload-socket\") pod \"660d18a9-a78d-49f8-8e10-537b4a257731\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " Oct 30 19:28:19.746244 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.746170 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/660d18a9-a78d-49f8-8e10-537b4a257731-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660d18a9-a78d-49f8-8e10-537b4a257731-istiod-ca-cert\") pod \"660d18a9-a78d-49f8-8e10-537b4a257731\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " Oct 30 19:28:19.746244 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.746199 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-workload-certs\") pod \"660d18a9-a78d-49f8-8e10-537b4a257731\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " Oct 30 19:28:19.746244 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.746225 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-istio-envoy\") pod \"660d18a9-a78d-49f8-8e10-537b4a257731\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " Oct 30 19:28:19.746611 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.746255 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-credential-socket\") pod \"660d18a9-a78d-49f8-8e10-537b4a257731\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " Oct 30 19:28:19.746611 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.746290 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660d18a9-a78d-49f8-8e10-537b4a257731-kube-api-access-9f67r\" (UniqueName: \"kubernetes.io/projected/660d18a9-a78d-49f8-8e10-537b4a257731-kube-api-access-9f67r\") pod \"660d18a9-a78d-49f8-8e10-537b4a257731\" (UID: \"660d18a9-a78d-49f8-8e10-537b4a257731\") " Oct 30 19:28:19.746611 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.746351 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-istio-data" pod "660d18a9-a78d-49f8-8e10-537b4a257731" (UID: "660d18a9-a78d-49f8-8e10-537b4a257731"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:19.746777 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.746656 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-workload-socket" pod "660d18a9-a78d-49f8-8e10-537b4a257731" (UID: "660d18a9-a78d-49f8-8e10-537b4a257731"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:19.746777 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.746694 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:28:19.746895 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.746875 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-credential-socket" pod "660d18a9-a78d-49f8-8e10-537b4a257731" (UID: "660d18a9-a78d-49f8-8e10-537b4a257731"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:19.747117 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.747045 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660d18a9-a78d-49f8-8e10-537b4a257731-istiod-ca-cert" pod "660d18a9-a78d-49f8-8e10-537b4a257731" (UID: "660d18a9-a78d-49f8-8e10-537b4a257731"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:28:19.747229 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.747198 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-workload-certs" pod "660d18a9-a78d-49f8-8e10-537b4a257731" (UID: "660d18a9-a78d-49f8-8e10-537b4a257731"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:19.749279 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.749246 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660d18a9-a78d-49f8-8e10-537b4a257731-istio-token" pod "660d18a9-a78d-49f8-8e10-537b4a257731" (UID: "660d18a9-a78d-49f8-8e10-537b4a257731"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:28:19.749412 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.749327 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/660d18a9-a78d-49f8-8e10-537b4a257731-istio-podinfo" pod "660d18a9-a78d-49f8-8e10-537b4a257731" (UID: "660d18a9-a78d-49f8-8e10-537b4a257731"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:28:19.749530 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.749511 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-istio-envoy" pod "660d18a9-a78d-49f8-8e10-537b4a257731" (UID: "660d18a9-a78d-49f8-8e10-537b4a257731"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:19.752275 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.752247 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660d18a9-a78d-49f8-8e10-537b4a257731-kube-api-access-9f67r" pod "660d18a9-a78d-49f8-8e10-537b4a257731" (UID: "660d18a9-a78d-49f8-8e10-537b4a257731"). InnerVolumeSpecName "kube-api-access-9f67r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:28:19.848421 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.848322 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660d18a9-a78d-49f8-8e10-537b4a257731-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:28:19.848421 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.848357 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660d18a9-a78d-49f8-8e10-537b4a257731-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:28:19.848421 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.848367 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:28:19.848421 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.848377 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660d18a9-a78d-49f8-8e10-537b4a257731-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:28:19.848421 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.848386 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:28:19.848421 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.848394 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:28:19.848421 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.848403 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660d18a9-a78d-49f8-8e10-537b4a257731-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:28:19.848421 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.848411 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f67r\" (UniqueName: \"kubernetes.io/projected/660d18a9-a78d-49f8-8e10-537b4a257731-kube-api-access-9f67r\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:28:19.860247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.860186 4531 generic.go:356] "Generic (PLEG): container finished" podID="660d18a9-a78d-49f8-8e10-537b4a257731" containerID="38c63315d96ea28e0551c2a21b38e602decdaf54bc2791faec85fc2e75d96b2d" exitCode=0 Oct 30 19:28:19.860525 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.860269 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" Oct 30 19:28:19.860525 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.860271 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" event={"ID":"660d18a9-a78d-49f8-8e10-537b4a257731","Type":"ContainerDied","Data":"38c63315d96ea28e0551c2a21b38e602decdaf54bc2791faec85fc2e75d96b2d"} Oct 30 19:28:19.860525 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.860360 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4" event={"ID":"660d18a9-a78d-49f8-8e10-537b4a257731","Type":"ContainerDied","Data":"59c9dab571b257f877313635faa76f2564f84b1534f2b5470e8c7bc515288960"} Oct 30 19:28:19.860525 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.860399 4531 scope.go:122] "RemoveContainer" containerID="38c63315d96ea28e0551c2a21b38e602decdaf54bc2791faec85fc2e75d96b2d" Oct 30 19:28:19.871850 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.871829 4531 scope.go:122] "RemoveContainer" containerID="38c63315d96ea28e0551c2a21b38e602decdaf54bc2791faec85fc2e75d96b2d" Oct 30 19:28:19.872293 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:28:19.872269 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38c63315d96ea28e0551c2a21b38e602decdaf54bc2791faec85fc2e75d96b2d\": container with ID starting with 38c63315d96ea28e0551c2a21b38e602decdaf54bc2791faec85fc2e75d96b2d not found: ID does not exist" containerID="38c63315d96ea28e0551c2a21b38e602decdaf54bc2791faec85fc2e75d96b2d" Oct 30 19:28:19.872391 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:19.872302 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38c63315d96ea28e0551c2a21b38e602decdaf54bc2791faec85fc2e75d96b2d"} err="failed to get container status \"38c63315d96ea28e0551c2a21b38e602decdaf54bc2791faec85fc2e75d96b2d\": rpc error: code = NotFound desc = could not find container \"38c63315d96ea28e0551c2a21b38e602decdaf54bc2791faec85fc2e75d96b2d\": container with ID starting with 38c63315d96ea28e0551c2a21b38e602decdaf54bc2791faec85fc2e75d96b2d not found: ID does not exist" Oct 30 19:28:20.028590 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:20.028539 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4"] Oct 30 19:28:20.033155 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:20.033103 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fd61e3-openshift-default-748b77c6b9-q2pz4"] Oct 30 19:28:20.102002 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:20.101897 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="660d18a9-a78d-49f8-8e10-537b4a257731" path="/var/lib/kubelet/pods/660d18a9-a78d-49f8-8e10-537b4a257731/volumes" Oct 30 19:28:26.400771 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:28:26.400731 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:28:56.398695 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:28:56.398652 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:28:57.397042 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:57.397003 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5"] Oct 30 19:28:57.446484 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:57.446446 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-81d66e-p6cs5"] Oct 30 19:28:58.099338 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:28:58.099290 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f87e49e-4c36-4df8-9c37-42537b5dc289" path="/var/lib/kubelet/pods/3f87e49e-4c36-4df8-9c37-42537b5dc289/volumes" Oct 30 19:29:02.765501 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:02.765457 4531 scope.go:122] "RemoveContainer" containerID="736eba678a20ccf0eb824f1f0a9381ee0ab268611a501a36c13c8a9d576b593d" Oct 30 19:29:18.377637 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:18.377599 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55b37bbe-gs8lv"] Oct 30 19:29:18.378090 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:18.378077 4531 cpu_manager.go:414] "containerMap: removing container" podUID="660d18a9-a78d-49f8-8e10-537b4a257731" containerName="istio-proxy" Oct 30 19:29:18.378158 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:18.378092 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660d18a9-a78d-49f8-8e10-537b4a257731" containerName="istio-proxy" Oct 30 19:29:18.378158 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:18.378142 4531 cpu_manager.go:414] "containerMap: removing container" podUID="cf335c92-b015-4e59-9142-181bd2e7aee1" containerName="revoke-keys" Oct 30 19:29:18.378158 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:18.378150 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf335c92-b015-4e59-9142-181bd2e7aee1" containerName="revoke-keys" Oct 30 19:29:18.378270 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:18.378161 4531 cpu_manager.go:414] "containerMap: removing container" podUID="b5ecb161-fdf5-4084-9080-0227a442d88a" containerName="istio-proxy" Oct 30 19:29:18.378270 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:18.378166 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5ecb161-fdf5-4084-9080-0227a442d88a" containerName="istio-proxy" Oct 30 19:29:18.378270 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:18.378250 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf335c92-b015-4e59-9142-181bd2e7aee1" containerName="revoke-keys" Oct 30 19:29:18.378270 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:18.378259 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5ecb161-fdf5-4084-9080-0227a442d88a" containerName="istio-proxy" Oct 30 19:29:18.378270 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:18.378267 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="660d18a9-a78d-49f8-8e10-537b4a257731" containerName="istio-proxy" Oct 30 19:29:18.383006 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:18.382988 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55b37bbe-gs8lv" Oct 30 19:29:18.438085 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:18.438010 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:29:18.487410 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:18.487375 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:29:18.487607 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:18.487448 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:29:18.487607 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:18.487453 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:29:18.489812 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:18.489791 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55b37bbe-gs8lv"] Oct 30 19:29:18.498083 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:18.498062 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n55l\" (UniqueName: \"kubernetes.io/projected/9a139411-416b-4beb-b089-fe85e87b485e-kube-api-access-5n55l\") pod \"maas-api-revoke-sub-55b37bbe-gs8lv\" (UID: \"9a139411-416b-4beb-b089-fe85e87b485e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55b37bbe-gs8lv" Oct 30 19:29:18.498174 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:18.498104 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a139411-416b-4beb-b089-fe85e87b485e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55b37bbe-gs8lv\" (UID: \"9a139411-416b-4beb-b089-fe85e87b485e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55b37bbe-gs8lv" Oct 30 19:29:18.498174 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:18.498159 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a139411-416b-4beb-b089-fe85e87b485e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55b37bbe-gs8lv\" (UID: \"9a139411-416b-4beb-b089-fe85e87b485e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55b37bbe-gs8lv" Oct 30 19:29:18.599550 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:18.599511 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n55l\" (UniqueName: \"kubernetes.io/projected/9a139411-416b-4beb-b089-fe85e87b485e-kube-api-access-5n55l\") pod \"maas-api-revoke-sub-55b37bbe-gs8lv\" (UID: \"9a139411-416b-4beb-b089-fe85e87b485e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55b37bbe-gs8lv" Oct 30 19:29:18.599772 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:18.599567 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a139411-416b-4beb-b089-fe85e87b485e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55b37bbe-gs8lv\" (UID: \"9a139411-416b-4beb-b089-fe85e87b485e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55b37bbe-gs8lv" Oct 30 19:29:18.599772 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:18.599622 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a139411-416b-4beb-b089-fe85e87b485e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55b37bbe-gs8lv\" (UID: \"9a139411-416b-4beb-b089-fe85e87b485e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55b37bbe-gs8lv" Oct 30 19:29:18.600274 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:18.600254 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a139411-416b-4beb-b089-fe85e87b485e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55b37bbe-gs8lv\" (UID: \"9a139411-416b-4beb-b089-fe85e87b485e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55b37bbe-gs8lv" Oct 30 19:29:18.602280 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:18.602258 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a139411-416b-4beb-b089-fe85e87b485e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55b37bbe-gs8lv\" (UID: \"9a139411-416b-4beb-b089-fe85e87b485e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55b37bbe-gs8lv" Oct 30 19:29:18.661860 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:18.661795 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n55l\" (UniqueName: \"kubernetes.io/projected/9a139411-416b-4beb-b089-fe85e87b485e-kube-api-access-5n55l\") pod \"maas-api-revoke-sub-55b37bbe-gs8lv\" (UID: \"9a139411-416b-4beb-b089-fe85e87b485e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55b37bbe-gs8lv" Oct 30 19:29:18.696386 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:18.696348 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55b37bbe-gs8lv" Oct 30 19:29:19.073622 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:19.073590 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55b37bbe-gs8lv"] Oct 30 19:29:19.075784 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:29:19.075751 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a139411_416b_4beb_b089_fe85e87b485e.slice/crio-079da32fcf6df8f2526d645fb5c1cb92645fb17550d984e0d1efff966bf43586 WatchSource:0}: Error finding container 079da32fcf6df8f2526d645fb5c1cb92645fb17550d984e0d1efff966bf43586: Status 404 returned error can't find the container with id 079da32fcf6df8f2526d645fb5c1cb92645fb17550d984e0d1efff966bf43586 Oct 30 19:29:19.199731 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:19.199692 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55b37bbe-gs8lv" event={"ID":"9a139411-416b-4beb-b089-fe85e87b485e","Type":"ContainerStarted","Data":"079da32fcf6df8f2526d645fb5c1cb92645fb17550d984e0d1efff966bf43586"} Oct 30 19:29:20.207572 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:20.207480 4531 generic.go:356] "Generic (PLEG): container finished" podID="9a139411-416b-4beb-b089-fe85e87b485e" containerID="1e9720ea88cce2c49a8eeab5260b93ae522dffdf4a97f45b530c821934d51b65" exitCode=0 Oct 30 19:29:20.208024 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:20.207569 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55b37bbe-gs8lv" event={"ID":"9a139411-416b-4beb-b089-fe85e87b485e","Type":"ContainerDied","Data":"1e9720ea88cce2c49a8eeab5260b93ae522dffdf4a97f45b530c821934d51b65"} Oct 30 19:29:21.378204 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:21.378181 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55b37bbe-gs8lv" Oct 30 19:29:21.536241 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:21.536209 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a139411-416b-4beb-b089-fe85e87b485e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a139411-416b-4beb-b089-fe85e87b485e-maas-api-openshift-ca\") pod \"9a139411-416b-4beb-b089-fe85e87b485e\" (UID: \"9a139411-416b-4beb-b089-fe85e87b485e\") " Oct 30 19:29:21.536241 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:21.536246 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a139411-416b-4beb-b089-fe85e87b485e-kube-api-access-5n55l\" (UniqueName: \"kubernetes.io/projected/9a139411-416b-4beb-b089-fe85e87b485e-kube-api-access-5n55l\") pod \"9a139411-416b-4beb-b089-fe85e87b485e\" (UID: \"9a139411-416b-4beb-b089-fe85e87b485e\") " Oct 30 19:29:21.536493 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:21.536364 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a139411-416b-4beb-b089-fe85e87b485e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a139411-416b-4beb-b089-fe85e87b485e-maas-api-xks-ca\") pod \"9a139411-416b-4beb-b089-fe85e87b485e\" (UID: \"9a139411-416b-4beb-b089-fe85e87b485e\") " Oct 30 19:29:21.536621 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:21.536594 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a139411-416b-4beb-b089-fe85e87b485e-maas-api-openshift-ca" pod "9a139411-416b-4beb-b089-fe85e87b485e" (UID: "9a139411-416b-4beb-b089-fe85e87b485e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:29:21.536859 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:21.536836 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a139411-416b-4beb-b089-fe85e87b485e-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:29:21.538670 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:21.538643 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a139411-416b-4beb-b089-fe85e87b485e-kube-api-access-5n55l" pod "9a139411-416b-4beb-b089-fe85e87b485e" (UID: "9a139411-416b-4beb-b089-fe85e87b485e"). InnerVolumeSpecName "kube-api-access-5n55l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:29:21.538670 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:21.538642 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a139411-416b-4beb-b089-fe85e87b485e-maas-api-xks-ca" pod "9a139411-416b-4beb-b089-fe85e87b485e" (UID: "9a139411-416b-4beb-b089-fe85e87b485e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:29:21.638117 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:21.638075 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n55l\" (UniqueName: \"kubernetes.io/projected/9a139411-416b-4beb-b089-fe85e87b485e-kube-api-access-5n55l\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:29:21.638117 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:21.638111 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a139411-416b-4beb-b089-fe85e87b485e-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:29:22.221815 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:22.221770 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55b37bbe-gs8lv" event={"ID":"9a139411-416b-4beb-b089-fe85e87b485e","Type":"ContainerDied","Data":"079da32fcf6df8f2526d645fb5c1cb92645fb17550d984e0d1efff966bf43586"} Oct 30 19:29:22.221815 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:22.221812 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55b37bbe-gs8lv" Oct 30 19:29:22.222056 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:22.221838 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="079da32fcf6df8f2526d645fb5c1cb92645fb17550d984e0d1efff966bf43586" Oct 30 19:29:26.399667 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:29:26.399627 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:29:29.176922 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:29.176883 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd1b3d8a-vr8tr"] Oct 30 19:29:29.177363 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:29.177355 4531 cpu_manager.go:414] "containerMap: removing container" podUID="9a139411-416b-4beb-b089-fe85e87b485e" containerName="revoke-keys" Oct 30 19:29:29.177409 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:29.177367 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a139411-416b-4beb-b089-fe85e87b485e" containerName="revoke-keys" Oct 30 19:29:29.177455 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:29.177445 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a139411-416b-4beb-b089-fe85e87b485e" containerName="revoke-keys" Oct 30 19:29:29.180757 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:29.180740 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1b3d8a-vr8tr" Oct 30 19:29:29.220765 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:29.220590 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edde5fa1-8c74-4755-b931-895636099038-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd1b3d8a-vr8tr\" (UID: \"edde5fa1-8c74-4755-b931-895636099038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1b3d8a-vr8tr" Oct 30 19:29:29.220765 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:29.220682 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edde5fa1-8c74-4755-b931-895636099038-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd1b3d8a-vr8tr\" (UID: \"edde5fa1-8c74-4755-b931-895636099038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1b3d8a-vr8tr" Oct 30 19:29:29.221052 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:29.220847 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkpkp\" (UniqueName: \"kubernetes.io/projected/edde5fa1-8c74-4755-b931-895636099038-kube-api-access-zkpkp\") pod \"maas-api-revoke-sub-dd1b3d8a-vr8tr\" (UID: \"edde5fa1-8c74-4755-b931-895636099038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1b3d8a-vr8tr" Oct 30 19:29:29.233120 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:29.233085 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:29:29.233120 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:29.233085 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:29:29.282705 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:29.282669 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:29:29.282929 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:29.282788 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:29:29.285217 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:29.285182 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd1b3d8a-vr8tr"] Oct 30 19:29:29.322374 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:29.322329 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkpkp\" (UniqueName: \"kubernetes.io/projected/edde5fa1-8c74-4755-b931-895636099038-kube-api-access-zkpkp\") pod \"maas-api-revoke-sub-dd1b3d8a-vr8tr\" (UID: \"edde5fa1-8c74-4755-b931-895636099038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1b3d8a-vr8tr" Oct 30 19:29:29.322614 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:29.322496 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edde5fa1-8c74-4755-b931-895636099038-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd1b3d8a-vr8tr\" (UID: \"edde5fa1-8c74-4755-b931-895636099038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1b3d8a-vr8tr" Oct 30 19:29:29.322614 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:29.322542 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edde5fa1-8c74-4755-b931-895636099038-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd1b3d8a-vr8tr\" (UID: \"edde5fa1-8c74-4755-b931-895636099038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1b3d8a-vr8tr" Oct 30 19:29:29.323242 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:29.323218 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edde5fa1-8c74-4755-b931-895636099038-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd1b3d8a-vr8tr\" (UID: \"edde5fa1-8c74-4755-b931-895636099038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1b3d8a-vr8tr" Oct 30 19:29:29.325400 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:29.325373 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edde5fa1-8c74-4755-b931-895636099038-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd1b3d8a-vr8tr\" (UID: \"edde5fa1-8c74-4755-b931-895636099038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1b3d8a-vr8tr" Oct 30 19:29:29.383635 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:29.383603 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkpkp\" (UniqueName: \"kubernetes.io/projected/edde5fa1-8c74-4755-b931-895636099038-kube-api-access-zkpkp\") pod \"maas-api-revoke-sub-dd1b3d8a-vr8tr\" (UID: \"edde5fa1-8c74-4755-b931-895636099038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1b3d8a-vr8tr" Oct 30 19:29:29.494425 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:29.494395 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1b3d8a-vr8tr" Oct 30 19:29:29.874659 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:29.874575 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd1b3d8a-vr8tr"] Oct 30 19:29:29.879886 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:29:29.879847 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedde5fa1_8c74_4755_b931_895636099038.slice/crio-0e9f23139559c4906890d3580e368c492ebe15816e2f5664e6e01a996c1ba13b WatchSource:0}: Error finding container 0e9f23139559c4906890d3580e368c492ebe15816e2f5664e6e01a996c1ba13b: Status 404 returned error can't find the container with id 0e9f23139559c4906890d3580e368c492ebe15816e2f5664e6e01a996c1ba13b Oct 30 19:29:30.269650 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:30.269613 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1b3d8a-vr8tr" event={"ID":"edde5fa1-8c74-4755-b931-895636099038","Type":"ContainerStarted","Data":"1b753196aef6f0182586f15a5c1f9e6b02b2b0479722702c41b7eb90614b0162"} Oct 30 19:29:30.270142 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:30.269674 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1b3d8a-vr8tr" event={"ID":"edde5fa1-8c74-4755-b931-895636099038","Type":"ContainerStarted","Data":"0e9f23139559c4906890d3580e368c492ebe15816e2f5664e6e01a996c1ba13b"} Oct 30 19:29:30.409235 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:30.409104 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1b3d8a-vr8tr" podStartSLOduration=1.409086151 podStartE2EDuration="1.409086151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:30.406192898 +0000 UTC m=+3330.433297856" watchObservedRunningTime="2026-10-30 19:29:30.409086151 +0000 UTC m=+3330.436191106" Oct 30 19:29:30.804088 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:30.804051 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4"] Oct 30 19:29:30.807878 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:30.807853 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4" Oct 30 19:29:30.840612 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:30.840572 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxf9x\" (UniqueName: \"kubernetes.io/projected/42119ccd-547b-41b3-95ac-9443ad52dd6f-kube-api-access-cxf9x\") pod \"maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4\" (UID: \"42119ccd-547b-41b3-95ac-9443ad52dd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4" Oct 30 19:29:30.840802 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:30.840671 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42119ccd-547b-41b3-95ac-9443ad52dd6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4\" (UID: \"42119ccd-547b-41b3-95ac-9443ad52dd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4" Oct 30 19:29:30.840802 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:30.840722 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42119ccd-547b-41b3-95ac-9443ad52dd6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4\" (UID: \"42119ccd-547b-41b3-95ac-9443ad52dd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4" Oct 30 19:29:30.867031 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:30.866998 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4"] Oct 30 19:29:30.942306 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:30.942275 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42119ccd-547b-41b3-95ac-9443ad52dd6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4\" (UID: \"42119ccd-547b-41b3-95ac-9443ad52dd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4" Oct 30 19:29:30.942508 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:30.942337 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42119ccd-547b-41b3-95ac-9443ad52dd6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4\" (UID: \"42119ccd-547b-41b3-95ac-9443ad52dd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4" Oct 30 19:29:30.942508 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:30.942402 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxf9x\" (UniqueName: \"kubernetes.io/projected/42119ccd-547b-41b3-95ac-9443ad52dd6f-kube-api-access-cxf9x\") pod \"maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4\" (UID: \"42119ccd-547b-41b3-95ac-9443ad52dd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4" Oct 30 19:29:30.943175 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:30.943158 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42119ccd-547b-41b3-95ac-9443ad52dd6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4\" (UID: \"42119ccd-547b-41b3-95ac-9443ad52dd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4" Oct 30 19:29:30.944892 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:30.944869 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42119ccd-547b-41b3-95ac-9443ad52dd6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4\" (UID: \"42119ccd-547b-41b3-95ac-9443ad52dd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4" Oct 30 19:29:31.003894 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:31.003862 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxf9x\" (UniqueName: \"kubernetes.io/projected/42119ccd-547b-41b3-95ac-9443ad52dd6f-kube-api-access-cxf9x\") pod \"maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4\" (UID: \"42119ccd-547b-41b3-95ac-9443ad52dd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4" Oct 30 19:29:31.169743 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:31.169650 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4" Oct 30 19:29:31.276963 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:31.276933 4531 generic.go:356] "Generic (PLEG): container finished" podID="edde5fa1-8c74-4755-b931-895636099038" containerID="1b753196aef6f0182586f15a5c1f9e6b02b2b0479722702c41b7eb90614b0162" exitCode=0 Oct 30 19:29:31.277480 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:31.277022 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1b3d8a-vr8tr" event={"ID":"edde5fa1-8c74-4755-b931-895636099038","Type":"ContainerDied","Data":"1b753196aef6f0182586f15a5c1f9e6b02b2b0479722702c41b7eb90614b0162"} Oct 30 19:29:31.547907 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:31.547881 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4"] Oct 30 19:29:31.550173 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:29:31.550122 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42119ccd_547b_41b3_95ac_9443ad52dd6f.slice/crio-fe4a11b09d273a897d3c2ed5d0cbd61fbc7a0a949b24c2f39cd979eaa8025ac1 WatchSource:0}: Error finding container fe4a11b09d273a897d3c2ed5d0cbd61fbc7a0a949b24c2f39cd979eaa8025ac1: Status 404 returned error can't find the container with id fe4a11b09d273a897d3c2ed5d0cbd61fbc7a0a949b24c2f39cd979eaa8025ac1 Oct 30 19:29:32.291591 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:32.291551 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4" event={"ID":"42119ccd-547b-41b3-95ac-9443ad52dd6f","Type":"ContainerStarted","Data":"a0775ecc3db7edbd332d094362e6a568668b157cd700620136269a9b47af1ee9"} Oct 30 19:29:32.292048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:32.291631 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4" event={"ID":"42119ccd-547b-41b3-95ac-9443ad52dd6f","Type":"ContainerStarted","Data":"fe4a11b09d273a897d3c2ed5d0cbd61fbc7a0a949b24c2f39cd979eaa8025ac1"} Oct 30 19:29:32.409855 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:32.409782 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4" podStartSLOduration=2.409763158 podStartE2EDuration="2.409763158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:32.407161467 +0000 UTC m=+3332.434266424" watchObservedRunningTime="2026-10-30 19:29:32.409763158 +0000 UTC m=+3332.436868117" Oct 30 19:29:32.451558 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:32.451526 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1b3d8a-vr8tr" Oct 30 19:29:32.560287 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:32.560208 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edde5fa1-8c74-4755-b931-895636099038-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edde5fa1-8c74-4755-b931-895636099038-maas-api-xks-ca\") pod \"edde5fa1-8c74-4755-b931-895636099038\" (UID: \"edde5fa1-8c74-4755-b931-895636099038\") " Oct 30 19:29:32.560287 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:32.560248 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edde5fa1-8c74-4755-b931-895636099038-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edde5fa1-8c74-4755-b931-895636099038-maas-api-openshift-ca\") pod \"edde5fa1-8c74-4755-b931-895636099038\" (UID: \"edde5fa1-8c74-4755-b931-895636099038\") " Oct 30 19:29:32.560287 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:32.560268 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edde5fa1-8c74-4755-b931-895636099038-kube-api-access-zkpkp\" (UniqueName: \"kubernetes.io/projected/edde5fa1-8c74-4755-b931-895636099038-kube-api-access-zkpkp\") pod \"edde5fa1-8c74-4755-b931-895636099038\" (UID: \"edde5fa1-8c74-4755-b931-895636099038\") " Oct 30 19:29:32.560647 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:32.560621 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edde5fa1-8c74-4755-b931-895636099038-maas-api-openshift-ca" pod "edde5fa1-8c74-4755-b931-895636099038" (UID: "edde5fa1-8c74-4755-b931-895636099038"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:29:32.562467 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:32.562444 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edde5fa1-8c74-4755-b931-895636099038-maas-api-xks-ca" pod "edde5fa1-8c74-4755-b931-895636099038" (UID: "edde5fa1-8c74-4755-b931-895636099038"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:29:32.562575 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:32.562519 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edde5fa1-8c74-4755-b931-895636099038-kube-api-access-zkpkp" pod "edde5fa1-8c74-4755-b931-895636099038" (UID: "edde5fa1-8c74-4755-b931-895636099038"). InnerVolumeSpecName "kube-api-access-zkpkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:29:32.663067 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:32.663028 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edde5fa1-8c74-4755-b931-895636099038-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:29:32.663067 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:32.663063 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edde5fa1-8c74-4755-b931-895636099038-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:29:32.663067 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:32.663074 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkpkp\" (UniqueName: \"kubernetes.io/projected/edde5fa1-8c74-4755-b931-895636099038-kube-api-access-zkpkp\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:29:33.298965 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:33.298938 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1b3d8a-vr8tr" Oct 30 19:29:33.298965 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:33.298955 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1b3d8a-vr8tr" event={"ID":"edde5fa1-8c74-4755-b931-895636099038","Type":"ContainerDied","Data":"0e9f23139559c4906890d3580e368c492ebe15816e2f5664e6e01a996c1ba13b"} Oct 30 19:29:33.299520 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:33.299009 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e9f23139559c4906890d3580e368c492ebe15816e2f5664e6e01a996c1ba13b" Oct 30 19:29:33.301349 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:33.301322 4531 generic.go:356] "Generic (PLEG): container finished" podID="42119ccd-547b-41b3-95ac-9443ad52dd6f" containerID="a0775ecc3db7edbd332d094362e6a568668b157cd700620136269a9b47af1ee9" exitCode=0 Oct 30 19:29:33.301498 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:33.301384 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4" event={"ID":"42119ccd-547b-41b3-95ac-9443ad52dd6f","Type":"ContainerDied","Data":"a0775ecc3db7edbd332d094362e6a568668b157cd700620136269a9b47af1ee9"} Oct 30 19:29:34.462642 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:34.462615 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4" Oct 30 19:29:34.582682 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:34.582597 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42119ccd-547b-41b3-95ac-9443ad52dd6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42119ccd-547b-41b3-95ac-9443ad52dd6f-maas-api-xks-ca\") pod \"42119ccd-547b-41b3-95ac-9443ad52dd6f\" (UID: \"42119ccd-547b-41b3-95ac-9443ad52dd6f\") " Oct 30 19:29:34.582858 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:34.582732 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42119ccd-547b-41b3-95ac-9443ad52dd6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42119ccd-547b-41b3-95ac-9443ad52dd6f-maas-api-openshift-ca\") pod \"42119ccd-547b-41b3-95ac-9443ad52dd6f\" (UID: \"42119ccd-547b-41b3-95ac-9443ad52dd6f\") " Oct 30 19:29:34.582858 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:34.582831 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42119ccd-547b-41b3-95ac-9443ad52dd6f-kube-api-access-cxf9x\" (UniqueName: \"kubernetes.io/projected/42119ccd-547b-41b3-95ac-9443ad52dd6f-kube-api-access-cxf9x\") pod \"42119ccd-547b-41b3-95ac-9443ad52dd6f\" (UID: \"42119ccd-547b-41b3-95ac-9443ad52dd6f\") " Oct 30 19:29:34.583104 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:34.583075 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42119ccd-547b-41b3-95ac-9443ad52dd6f-maas-api-openshift-ca" pod "42119ccd-547b-41b3-95ac-9443ad52dd6f" (UID: "42119ccd-547b-41b3-95ac-9443ad52dd6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:29:34.585207 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:34.585170 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42119ccd-547b-41b3-95ac-9443ad52dd6f-kube-api-access-cxf9x" pod "42119ccd-547b-41b3-95ac-9443ad52dd6f" (UID: "42119ccd-547b-41b3-95ac-9443ad52dd6f"). InnerVolumeSpecName "kube-api-access-cxf9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:29:34.585438 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:34.585414 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42119ccd-547b-41b3-95ac-9443ad52dd6f-maas-api-xks-ca" pod "42119ccd-547b-41b3-95ac-9443ad52dd6f" (UID: "42119ccd-547b-41b3-95ac-9443ad52dd6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:29:34.684172 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:34.684121 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxf9x\" (UniqueName: \"kubernetes.io/projected/42119ccd-547b-41b3-95ac-9443ad52dd6f-kube-api-access-cxf9x\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:29:34.684172 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:34.684167 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42119ccd-547b-41b3-95ac-9443ad52dd6f-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:29:34.684172 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:34.684178 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42119ccd-547b-41b3-95ac-9443ad52dd6f-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:29:35.313240 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:35.313209 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4" Oct 30 19:29:35.313447 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:35.313206 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4" event={"ID":"42119ccd-547b-41b3-95ac-9443ad52dd6f","Type":"ContainerDied","Data":"fe4a11b09d273a897d3c2ed5d0cbd61fbc7a0a949b24c2f39cd979eaa8025ac1"} Oct 30 19:29:35.313447 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:35.313345 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe4a11b09d273a897d3c2ed5d0cbd61fbc7a0a949b24c2f39cd979eaa8025ac1" Oct 30 19:29:42.328739 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:42.328701 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf"] Oct 30 19:29:42.436319 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:42.436284 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7"] Oct 30 19:29:47.329757 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:47.329716 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf" podUID="bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890" containerName="payload-processing" containerID="cri-o://d2c268047fbbd9e02012b6d15aa92aa53f16d4de7e0f7df3b274bddca7678dde" gracePeriod=25 Oct 30 19:29:47.436945 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:47.436880 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7" podUID="865f93c1-d33c-4231-99ff-79bcf64acaca" containerName="payload-pre-processing" containerID="cri-o://aca4c2e863728db8a0e7f43929b5435d8139adf3a0f03958e7917be5e4fa2936" gracePeriod=25 Oct 30 19:29:47.606735 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:47.606705 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf" Oct 30 19:29:47.695564 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:47.695538 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7" Oct 30 19:29:47.729714 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:47.729683 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890-kube-api-access-kndqj\" (UniqueName: \"kubernetes.io/projected/bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890-kube-api-access-kndqj\") pod \"bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890\" (UID: \"bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890\") " Oct 30 19:29:47.729892 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:47.729724 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890-plugins-config-volume\") pod \"bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890\" (UID: \"bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890\") " Oct 30 19:29:47.730208 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:47.730176 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890-plugins-config-volume" pod "bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890" (UID: "bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:29:47.732461 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:47.732438 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890-kube-api-access-kndqj" pod "bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890" (UID: "bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890"). InnerVolumeSpecName "kube-api-access-kndqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:29:47.830995 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:47.830954 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/865f93c1-d33c-4231-99ff-79bcf64acaca-kube-api-access-7rbvd\" (UniqueName: \"kubernetes.io/projected/865f93c1-d33c-4231-99ff-79bcf64acaca-kube-api-access-7rbvd\") pod \"865f93c1-d33c-4231-99ff-79bcf64acaca\" (UID: \"865f93c1-d33c-4231-99ff-79bcf64acaca\") " Oct 30 19:29:47.831235 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:47.831080 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/865f93c1-d33c-4231-99ff-79bcf64acaca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/865f93c1-d33c-4231-99ff-79bcf64acaca-plugins-config-volume\") pod \"865f93c1-d33c-4231-99ff-79bcf64acaca\" (UID: \"865f93c1-d33c-4231-99ff-79bcf64acaca\") " Oct 30 19:29:47.831441 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:47.831416 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/865f93c1-d33c-4231-99ff-79bcf64acaca-plugins-config-volume" pod "865f93c1-d33c-4231-99ff-79bcf64acaca" (UID: "865f93c1-d33c-4231-99ff-79bcf64acaca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:29:47.831441 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:47.831431 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kndqj\" (UniqueName: \"kubernetes.io/projected/bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890-kube-api-access-kndqj\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:29:47.831547 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:47.831451 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:29:47.833253 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:47.833209 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/865f93c1-d33c-4231-99ff-79bcf64acaca-kube-api-access-7rbvd" pod "865f93c1-d33c-4231-99ff-79bcf64acaca" (UID: "865f93c1-d33c-4231-99ff-79bcf64acaca"). InnerVolumeSpecName "kube-api-access-7rbvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:29:47.932075 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:47.932030 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rbvd\" (UniqueName: \"kubernetes.io/projected/865f93c1-d33c-4231-99ff-79bcf64acaca-kube-api-access-7rbvd\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:29:47.932075 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:47.932069 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/865f93c1-d33c-4231-99ff-79bcf64acaca-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:29:48.390476 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:48.390402 4531 generic.go:356] "Generic (PLEG): container finished" podID="bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890" containerID="d2c268047fbbd9e02012b6d15aa92aa53f16d4de7e0f7df3b274bddca7678dde" exitCode=0 Oct 30 19:29:48.390476 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:48.390458 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf" Oct 30 19:29:48.390968 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:48.390481 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf" event={"ID":"bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890","Type":"ContainerDied","Data":"d2c268047fbbd9e02012b6d15aa92aa53f16d4de7e0f7df3b274bddca7678dde"} Oct 30 19:29:48.390968 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:48.390537 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf" event={"ID":"bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890","Type":"ContainerDied","Data":"0b11db2209218bad4e27898ea41ff6bbd792116e6c1637db29dd3ad51c6af991"} Oct 30 19:29:48.390968 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:48.390563 4531 scope.go:122] "RemoveContainer" containerID="d2c268047fbbd9e02012b6d15aa92aa53f16d4de7e0f7df3b274bddca7678dde" Oct 30 19:29:48.391967 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:48.391947 4531 generic.go:356] "Generic (PLEG): container finished" podID="865f93c1-d33c-4231-99ff-79bcf64acaca" containerID="aca4c2e863728db8a0e7f43929b5435d8139adf3a0f03958e7917be5e4fa2936" exitCode=0 Oct 30 19:29:48.392071 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:48.392020 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7" Oct 30 19:29:48.392071 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:48.392032 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7" event={"ID":"865f93c1-d33c-4231-99ff-79bcf64acaca","Type":"ContainerDied","Data":"aca4c2e863728db8a0e7f43929b5435d8139adf3a0f03958e7917be5e4fa2936"} Oct 30 19:29:48.392198 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:48.392092 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7" event={"ID":"865f93c1-d33c-4231-99ff-79bcf64acaca","Type":"ContainerDied","Data":"df37e00953069aad1a074a9b71f5e832006d1bacbd9f0e6631093c741ecb7ea9"} Oct 30 19:29:48.400030 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:48.400010 4531 scope.go:122] "RemoveContainer" containerID="d2c268047fbbd9e02012b6d15aa92aa53f16d4de7e0f7df3b274bddca7678dde" Oct 30 19:29:48.400295 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:29:48.400276 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2c268047fbbd9e02012b6d15aa92aa53f16d4de7e0f7df3b274bddca7678dde\": container with ID starting with d2c268047fbbd9e02012b6d15aa92aa53f16d4de7e0f7df3b274bddca7678dde not found: ID does not exist" containerID="d2c268047fbbd9e02012b6d15aa92aa53f16d4de7e0f7df3b274bddca7678dde" Oct 30 19:29:48.400362 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:48.400303 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2c268047fbbd9e02012b6d15aa92aa53f16d4de7e0f7df3b274bddca7678dde"} err="failed to get container status \"d2c268047fbbd9e02012b6d15aa92aa53f16d4de7e0f7df3b274bddca7678dde\": rpc error: code = NotFound desc = could not find container \"d2c268047fbbd9e02012b6d15aa92aa53f16d4de7e0f7df3b274bddca7678dde\": container with ID starting with d2c268047fbbd9e02012b6d15aa92aa53f16d4de7e0f7df3b274bddca7678dde not found: ID does not exist" Oct 30 19:29:48.400362 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:48.400321 4531 scope.go:122] "RemoveContainer" containerID="aca4c2e863728db8a0e7f43929b5435d8139adf3a0f03958e7917be5e4fa2936" Oct 30 19:29:48.408803 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:48.408787 4531 scope.go:122] "RemoveContainer" containerID="aca4c2e863728db8a0e7f43929b5435d8139adf3a0f03958e7917be5e4fa2936" Oct 30 19:29:48.409059 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:29:48.409040 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aca4c2e863728db8a0e7f43929b5435d8139adf3a0f03958e7917be5e4fa2936\": container with ID starting with aca4c2e863728db8a0e7f43929b5435d8139adf3a0f03958e7917be5e4fa2936 not found: ID does not exist" containerID="aca4c2e863728db8a0e7f43929b5435d8139adf3a0f03958e7917be5e4fa2936" Oct 30 19:29:48.409205 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:48.409069 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aca4c2e863728db8a0e7f43929b5435d8139adf3a0f03958e7917be5e4fa2936"} err="failed to get container status \"aca4c2e863728db8a0e7f43929b5435d8139adf3a0f03958e7917be5e4fa2936\": rpc error: code = NotFound desc = could not find container \"aca4c2e863728db8a0e7f43929b5435d8139adf3a0f03958e7917be5e4fa2936\": container with ID starting with aca4c2e863728db8a0e7f43929b5435d8139adf3a0f03958e7917be5e4fa2936 not found: ID does not exist" Oct 30 19:29:48.556496 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:48.556458 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf"] Oct 30 19:29:48.606210 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:48.606165 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f5dfb8-5b6dbc97cb-2htpf"] Oct 30 19:29:48.670029 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:48.669932 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7"] Oct 30 19:29:48.675603 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:48.675567 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f5dfb8-5f9578ddff-jpvw7"] Oct 30 19:29:50.098346 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:50.098312 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="865f93c1-d33c-4231-99ff-79bcf64acaca" path="/var/lib/kubelet/pods/865f93c1-d33c-4231-99ff-79bcf64acaca/volumes" Oct 30 19:29:50.098723 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:29:50.098674 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890" path="/var/lib/kubelet/pods/bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890/volumes" Oct 30 19:29:56.399340 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:29:56.399293 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:30:00.173309 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.173279 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt"] Oct 30 19:30:00.173813 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.173709 4531 cpu_manager.go:414] "containerMap: removing container" podUID="865f93c1-d33c-4231-99ff-79bcf64acaca" containerName="payload-pre-processing" Oct 30 19:30:00.173813 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.173721 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="865f93c1-d33c-4231-99ff-79bcf64acaca" containerName="payload-pre-processing" Oct 30 19:30:00.173813 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.173737 4531 cpu_manager.go:414] "containerMap: removing container" podUID="bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890" containerName="payload-processing" Oct 30 19:30:00.173813 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.173743 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890" containerName="payload-processing" Oct 30 19:30:00.173813 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.173755 4531 cpu_manager.go:414] "containerMap: removing container" podUID="42119ccd-547b-41b3-95ac-9443ad52dd6f" containerName="revoke-keys" Oct 30 19:30:00.173813 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.173760 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42119ccd-547b-41b3-95ac-9443ad52dd6f" containerName="revoke-keys" Oct 30 19:30:00.173813 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.173766 4531 cpu_manager.go:414] "containerMap: removing container" podUID="edde5fa1-8c74-4755-b931-895636099038" containerName="revoke-keys" Oct 30 19:30:00.173813 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.173771 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edde5fa1-8c74-4755-b931-895636099038" containerName="revoke-keys" Oct 30 19:30:00.174082 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.173846 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="865f93c1-d33c-4231-99ff-79bcf64acaca" containerName="payload-pre-processing" Oct 30 19:30:00.174082 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.173854 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb4d1d0b-5ca8-4cce-9e94-df6f2dd71890" containerName="payload-processing" Oct 30 19:30:00.174082 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.173862 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="42119ccd-547b-41b3-95ac-9443ad52dd6f" containerName="revoke-keys" Oct 30 19:30:00.174082 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.173869 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="edde5fa1-8c74-4755-b931-895636099038" containerName="revoke-keys" Oct 30 19:30:00.178364 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.178342 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6"] Oct 30 19:30:00.178510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.178497 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt" Oct 30 19:30:00.181679 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.181662 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9"] Oct 30 19:30:00.181786 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.181763 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6" Oct 30 19:30:00.185050 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.185032 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9" Oct 30 19:30:00.223216 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.223187 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg"] Oct 30 19:30:00.226633 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.226613 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l"] Oct 30 19:30:00.226836 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.226802 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg" Oct 30 19:30:00.230380 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.230362 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb"] Oct 30 19:30:00.230489 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.230467 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l" Oct 30 19:30:00.233775 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.233724 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w"] Oct 30 19:30:00.233892 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.233835 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb" Oct 30 19:30:00.237468 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.237447 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7"] Oct 30 19:30:00.237606 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.237578 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w" Oct 30 19:30:00.241713 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.241691 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9"] Oct 30 19:30:00.241823 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.241809 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7" Oct 30 19:30:00.245140 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.245113 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9" Oct 30 19:30:00.266446 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.266418 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqcps\" (UniqueName: \"kubernetes.io/projected/747b02b0-b487-4542-9174-862708674b4c-kube-api-access-vqcps\") pod \"maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l\" (UID: \"747b02b0-b487-4542-9174-862708674b4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l" Oct 30 19:30:00.266613 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.266451 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/747b02b0-b487-4542-9174-862708674b4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l\" (UID: \"747b02b0-b487-4542-9174-862708674b4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l" Oct 30 19:30:00.266613 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.266472 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfmpg\" (UniqueName: \"kubernetes.io/projected/8d3ce71a-6c7f-4aca-829e-10b51893d417-kube-api-access-lfmpg\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb\" (UID: \"8d3ce71a-6c7f-4aca-829e-10b51893d417\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb" Oct 30 19:30:00.266613 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.266492 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10065963-ac42-42d0-902e-3ec7e87ebd06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt\" (UID: \"10065963-ac42-42d0-902e-3ec7e87ebd06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt" Oct 30 19:30:00.266790 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.266621 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5972ca80-ec09-4b6a-be76-847b415f88fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7\" (UID: \"5972ca80-ec09-4b6a-be76-847b415f88fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7" Oct 30 19:30:00.266790 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.266673 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f45f6f0-a99e-4381-a553-270bdb2f7621-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9\" (UID: \"6f45f6f0-a99e-4381-a553-270bdb2f7621\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9" Oct 30 19:30:00.266790 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.266718 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hv7m\" (UniqueName: \"kubernetes.io/projected/10065963-ac42-42d0-902e-3ec7e87ebd06-kube-api-access-7hv7m\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt\" (UID: \"10065963-ac42-42d0-902e-3ec7e87ebd06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt" Oct 30 19:30:00.266790 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.266754 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4cnv\" (UniqueName: \"kubernetes.io/projected/6f45f6f0-a99e-4381-a553-270bdb2f7621-kube-api-access-r4cnv\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9\" (UID: \"6f45f6f0-a99e-4381-a553-270bdb2f7621\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9" Oct 30 19:30:00.266978 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.266826 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a549c34-f860-401d-a5e6-9c45f015cb54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w\" (UID: \"0a549c34-f860-401d-a5e6-9c45f015cb54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w" Oct 30 19:30:00.266978 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.266949 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15436c31-541b-4441-b020-dc795e1387aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9\" (UID: \"15436c31-541b-4441-b020-dc795e1387aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9" Oct 30 19:30:00.267065 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.267046 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3ce71a-6c7f-4aca-829e-10b51893d417-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb\" (UID: \"8d3ce71a-6c7f-4aca-829e-10b51893d417\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb" Oct 30 19:30:00.267114 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.267100 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f6ffce-2030-4be2-bf0a-cf4a56dec691-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg\" (UID: \"65f6ffce-2030-4be2-bf0a-cf4a56dec691\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg" Oct 30 19:30:00.267217 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.267201 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsshr\" (UniqueName: \"kubernetes.io/projected/0a549c34-f860-401d-a5e6-9c45f015cb54-kube-api-access-wsshr\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w\" (UID: \"0a549c34-f860-401d-a5e6-9c45f015cb54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w" Oct 30 19:30:00.267259 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.267231 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dz76\" (UniqueName: \"kubernetes.io/projected/5972ca80-ec09-4b6a-be76-847b415f88fb-kube-api-access-8dz76\") pod \"maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7\" (UID: \"5972ca80-ec09-4b6a-be76-847b415f88fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7" Oct 30 19:30:00.267301 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.267290 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp9kt\" (UniqueName: \"kubernetes.io/projected/15436c31-541b-4441-b020-dc795e1387aa-kube-api-access-rp9kt\") pod \"maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9\" (UID: \"15436c31-541b-4441-b020-dc795e1387aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9" Oct 30 19:30:00.267393 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.267309 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gfw9\" (UniqueName: \"kubernetes.io/projected/541165d3-33cc-415c-9932-75e6502c70a8-kube-api-access-4gfw9\") pod \"maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6\" (UID: \"541165d3-33cc-415c-9932-75e6502c70a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6" Oct 30 19:30:00.267393 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.267337 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/541165d3-33cc-415c-9932-75e6502c70a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6\" (UID: \"541165d3-33cc-415c-9932-75e6502c70a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6" Oct 30 19:30:00.267494 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.267436 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5gvt\" (UniqueName: \"kubernetes.io/projected/65f6ffce-2030-4be2-bf0a-cf4a56dec691-kube-api-access-x5gvt\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg\" (UID: \"65f6ffce-2030-4be2-bf0a-cf4a56dec691\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg" Oct 30 19:30:00.270158 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.270116 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:30:00.270158 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.270116 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:30:00.270372 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.270194 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:30:00.275225 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.275195 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt"] Oct 30 19:30:00.278080 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.278053 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9"] Oct 30 19:30:00.280286 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.280265 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6"] Oct 30 19:30:00.326527 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.326485 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg"] Oct 30 19:30:00.331284 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.331247 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb"] Oct 30 19:30:00.336246 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.336210 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w"] Oct 30 19:30:00.368772 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.368733 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f6ffce-2030-4be2-bf0a-cf4a56dec691-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg\" (UID: \"65f6ffce-2030-4be2-bf0a-cf4a56dec691\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg" Oct 30 19:30:00.368976 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.368817 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsshr\" (UniqueName: \"kubernetes.io/projected/0a549c34-f860-401d-a5e6-9c45f015cb54-kube-api-access-wsshr\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w\" (UID: \"0a549c34-f860-401d-a5e6-9c45f015cb54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w" Oct 30 19:30:00.368976 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.368854 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dz76\" (UniqueName: \"kubernetes.io/projected/5972ca80-ec09-4b6a-be76-847b415f88fb-kube-api-access-8dz76\") pod \"maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7\" (UID: \"5972ca80-ec09-4b6a-be76-847b415f88fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7" Oct 30 19:30:00.368976 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.368887 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp9kt\" (UniqueName: \"kubernetes.io/projected/15436c31-541b-4441-b020-dc795e1387aa-kube-api-access-rp9kt\") pod \"maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9\" (UID: \"15436c31-541b-4441-b020-dc795e1387aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9" Oct 30 19:30:00.368976 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.368915 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gfw9\" (UniqueName: \"kubernetes.io/projected/541165d3-33cc-415c-9932-75e6502c70a8-kube-api-access-4gfw9\") pod \"maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6\" (UID: \"541165d3-33cc-415c-9932-75e6502c70a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6" Oct 30 19:30:00.368976 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.368953 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/541165d3-33cc-415c-9932-75e6502c70a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6\" (UID: \"541165d3-33cc-415c-9932-75e6502c70a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6" Oct 30 19:30:00.369280 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.369008 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5gvt\" (UniqueName: \"kubernetes.io/projected/65f6ffce-2030-4be2-bf0a-cf4a56dec691-kube-api-access-x5gvt\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg\" (UID: \"65f6ffce-2030-4be2-bf0a-cf4a56dec691\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg" Oct 30 19:30:00.369280 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.369058 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqcps\" (UniqueName: \"kubernetes.io/projected/747b02b0-b487-4542-9174-862708674b4c-kube-api-access-vqcps\") pod \"maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l\" (UID: \"747b02b0-b487-4542-9174-862708674b4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l" Oct 30 19:30:00.369280 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.369085 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/747b02b0-b487-4542-9174-862708674b4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l\" (UID: \"747b02b0-b487-4542-9174-862708674b4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l" Oct 30 19:30:00.369280 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.369111 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfmpg\" (UniqueName: \"kubernetes.io/projected/8d3ce71a-6c7f-4aca-829e-10b51893d417-kube-api-access-lfmpg\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb\" (UID: \"8d3ce71a-6c7f-4aca-829e-10b51893d417\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb" Oct 30 19:30:00.369280 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.369179 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10065963-ac42-42d0-902e-3ec7e87ebd06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt\" (UID: \"10065963-ac42-42d0-902e-3ec7e87ebd06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt" Oct 30 19:30:00.369280 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.369275 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5972ca80-ec09-4b6a-be76-847b415f88fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7\" (UID: \"5972ca80-ec09-4b6a-be76-847b415f88fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7" Oct 30 19:30:00.369624 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.369322 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f45f6f0-a99e-4381-a553-270bdb2f7621-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9\" (UID: \"6f45f6f0-a99e-4381-a553-270bdb2f7621\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9" Oct 30 19:30:00.369624 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.369347 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hv7m\" (UniqueName: \"kubernetes.io/projected/10065963-ac42-42d0-902e-3ec7e87ebd06-kube-api-access-7hv7m\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt\" (UID: \"10065963-ac42-42d0-902e-3ec7e87ebd06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt" Oct 30 19:30:00.369624 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.369395 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4cnv\" (UniqueName: \"kubernetes.io/projected/6f45f6f0-a99e-4381-a553-270bdb2f7621-kube-api-access-r4cnv\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9\" (UID: \"6f45f6f0-a99e-4381-a553-270bdb2f7621\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9" Oct 30 19:30:00.369624 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.369426 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a549c34-f860-401d-a5e6-9c45f015cb54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w\" (UID: \"0a549c34-f860-401d-a5e6-9c45f015cb54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w" Oct 30 19:30:00.369624 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.369458 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15436c31-541b-4441-b020-dc795e1387aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9\" (UID: \"15436c31-541b-4441-b020-dc795e1387aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9" Oct 30 19:30:00.369624 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.369554 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3ce71a-6c7f-4aca-829e-10b51893d417-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb\" (UID: \"8d3ce71a-6c7f-4aca-829e-10b51893d417\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb" Oct 30 19:30:00.369950 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.369654 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f6ffce-2030-4be2-bf0a-cf4a56dec691-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg\" (UID: \"65f6ffce-2030-4be2-bf0a-cf4a56dec691\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg" Oct 30 19:30:00.369950 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.369895 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10065963-ac42-42d0-902e-3ec7e87ebd06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt\" (UID: \"10065963-ac42-42d0-902e-3ec7e87ebd06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt" Oct 30 19:30:00.370110 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.370047 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/541165d3-33cc-415c-9932-75e6502c70a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6\" (UID: \"541165d3-33cc-415c-9932-75e6502c70a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6" Oct 30 19:30:00.370399 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.370370 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3ce71a-6c7f-4aca-829e-10b51893d417-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb\" (UID: \"8d3ce71a-6c7f-4aca-829e-10b51893d417\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb" Oct 30 19:30:00.370732 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.370706 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/747b02b0-b487-4542-9174-862708674b4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l\" (UID: \"747b02b0-b487-4542-9174-862708674b4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l" Oct 30 19:30:00.370836 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.370777 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5972ca80-ec09-4b6a-be76-847b415f88fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7\" (UID: \"5972ca80-ec09-4b6a-be76-847b415f88fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7" Oct 30 19:30:00.371269 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.371243 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f45f6f0-a99e-4381-a553-270bdb2f7621-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9\" (UID: \"6f45f6f0-a99e-4381-a553-270bdb2f7621\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9" Oct 30 19:30:00.371380 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.371329 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a549c34-f860-401d-a5e6-9c45f015cb54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w\" (UID: \"0a549c34-f860-401d-a5e6-9c45f015cb54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w" Oct 30 19:30:00.371430 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.371371 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15436c31-541b-4441-b020-dc795e1387aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9\" (UID: \"15436c31-541b-4441-b020-dc795e1387aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9" Oct 30 19:30:00.383407 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.383251 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7"] Oct 30 19:30:00.404104 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.402263 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l"] Oct 30 19:30:00.404953 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.404919 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9"] Oct 30 19:30:00.434349 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.434274 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqcps\" (UniqueName: \"kubernetes.io/projected/747b02b0-b487-4542-9174-862708674b4c-kube-api-access-vqcps\") pod \"maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l\" (UID: \"747b02b0-b487-4542-9174-862708674b4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l" Oct 30 19:30:00.434349 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.434316 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsshr\" (UniqueName: \"kubernetes.io/projected/0a549c34-f860-401d-a5e6-9c45f015cb54-kube-api-access-wsshr\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w\" (UID: \"0a549c34-f860-401d-a5e6-9c45f015cb54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w" Oct 30 19:30:00.434594 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.434570 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hv7m\" (UniqueName: \"kubernetes.io/projected/10065963-ac42-42d0-902e-3ec7e87ebd06-kube-api-access-7hv7m\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt\" (UID: \"10065963-ac42-42d0-902e-3ec7e87ebd06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt" Oct 30 19:30:00.438243 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.438194 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfmpg\" (UniqueName: \"kubernetes.io/projected/8d3ce71a-6c7f-4aca-829e-10b51893d417-kube-api-access-lfmpg\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb\" (UID: \"8d3ce71a-6c7f-4aca-829e-10b51893d417\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb" Oct 30 19:30:00.438604 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.438582 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5gvt\" (UniqueName: \"kubernetes.io/projected/65f6ffce-2030-4be2-bf0a-cf4a56dec691-kube-api-access-x5gvt\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg\" (UID: \"65f6ffce-2030-4be2-bf0a-cf4a56dec691\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg" Oct 30 19:30:00.438687 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.438609 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp9kt\" (UniqueName: \"kubernetes.io/projected/15436c31-541b-4441-b020-dc795e1387aa-kube-api-access-rp9kt\") pod \"maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9\" (UID: \"15436c31-541b-4441-b020-dc795e1387aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9" Oct 30 19:30:00.438687 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.438671 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dz76\" (UniqueName: \"kubernetes.io/projected/5972ca80-ec09-4b6a-be76-847b415f88fb-kube-api-access-8dz76\") pod \"maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7\" (UID: \"5972ca80-ec09-4b6a-be76-847b415f88fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7" Oct 30 19:30:00.438687 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.438674 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gfw9\" (UniqueName: \"kubernetes.io/projected/541165d3-33cc-415c-9932-75e6502c70a8-kube-api-access-4gfw9\") pod \"maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6\" (UID: \"541165d3-33cc-415c-9932-75e6502c70a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6" Oct 30 19:30:00.439153 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.439110 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4cnv\" (UniqueName: \"kubernetes.io/projected/6f45f6f0-a99e-4381-a553-270bdb2f7621-kube-api-access-r4cnv\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9\" (UID: \"6f45f6f0-a99e-4381-a553-270bdb2f7621\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9" Oct 30 19:30:00.494752 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.494713 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt" Oct 30 19:30:00.502310 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.502282 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6" Oct 30 19:30:00.508473 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.508439 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9" Oct 30 19:30:00.542867 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.541023 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg" Oct 30 19:30:00.549158 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.548429 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l" Oct 30 19:30:00.555993 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.555968 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb" Oct 30 19:30:00.568593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.567960 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w" Oct 30 19:30:00.621225 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.575505 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7" Oct 30 19:30:00.621225 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.582162 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9" Oct 30 19:30:00.902632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:00.902602 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt"] Oct 30 19:30:00.907063 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:30:00.907030 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10065963_ac42_42d0_902e_3ec7e87ebd06.slice/crio-4bbb8506aed02daf18001c76d1f736ea95cb43b2b89e3f29712890e2de038637 WatchSource:0}: Error finding container 4bbb8506aed02daf18001c76d1f736ea95cb43b2b89e3f29712890e2de038637: Status 404 returned error can't find the container with id 4bbb8506aed02daf18001c76d1f736ea95cb43b2b89e3f29712890e2de038637 Oct 30 19:30:01.041027 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:01.040992 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l"] Oct 30 19:30:01.051756 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:30:01.051727 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747b02b0_b487_4542_9174_862708674b4c.slice/crio-3e087372270407edf343498ef145c5589863e08bb4b9b178d4087dae73bccb4e WatchSource:0}: Error finding container 3e087372270407edf343498ef145c5589863e08bb4b9b178d4087dae73bccb4e: Status 404 returned error can't find the container with id 3e087372270407edf343498ef145c5589863e08bb4b9b178d4087dae73bccb4e Oct 30 19:30:01.061817 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:01.061796 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w"] Oct 30 19:30:01.064798 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:30:01.064768 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a549c34_f860_401d_a5e6_9c45f015cb54.slice/crio-75cb69c01ff29b22547af0a8fc9bdc96dfbb3316d2d29911adfb23447ba40eac WatchSource:0}: Error finding container 75cb69c01ff29b22547af0a8fc9bdc96dfbb3316d2d29911adfb23447ba40eac: Status 404 returned error can't find the container with id 75cb69c01ff29b22547af0a8fc9bdc96dfbb3316d2d29911adfb23447ba40eac Oct 30 19:30:01.070997 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:01.070971 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb"] Oct 30 19:30:01.082346 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:30:01.082254 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d3ce71a_6c7f_4aca_829e_10b51893d417.slice/crio-e514fb251de854c2d073b36e3cdd152f1368a57793dcb925e1fe6aae008dc5b1 WatchSource:0}: Error finding container e514fb251de854c2d073b36e3cdd152f1368a57793dcb925e1fe6aae008dc5b1: Status 404 returned error can't find the container with id e514fb251de854c2d073b36e3cdd152f1368a57793dcb925e1fe6aae008dc5b1 Oct 30 19:30:01.090700 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:01.090267 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7"] Oct 30 19:30:01.093070 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:01.093050 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9"] Oct 30 19:30:01.094890 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:30:01.094865 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5972ca80_ec09_4b6a_be76_847b415f88fb.slice/crio-0f3c7961e462fafc7891f288db6274a4a19c2b7be3be8a3176182ff9d911be52 WatchSource:0}: Error finding container 0f3c7961e462fafc7891f288db6274a4a19c2b7be3be8a3176182ff9d911be52: Status 404 returned error can't find the container with id 0f3c7961e462fafc7891f288db6274a4a19c2b7be3be8a3176182ff9d911be52 Oct 30 19:30:01.095292 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:01.095274 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9"] Oct 30 19:30:01.095610 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:30:01.095586 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15436c31_541b_4441_b020_dc795e1387aa.slice/crio-30eb3363637d8dde29ef6fd1598c9d94edfe5f66fb29e2091dac7889b4a41c3b WatchSource:0}: Error finding container 30eb3363637d8dde29ef6fd1598c9d94edfe5f66fb29e2091dac7889b4a41c3b: Status 404 returned error can't find the container with id 30eb3363637d8dde29ef6fd1598c9d94edfe5f66fb29e2091dac7889b4a41c3b Oct 30 19:30:01.105234 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:30:01.105207 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f45f6f0_a99e_4381_a553_270bdb2f7621.slice/crio-dc7fa253b5e0125fdadd5a72892bde7533d867006bff71a3bcf93c0d608c95f4 WatchSource:0}: Error finding container dc7fa253b5e0125fdadd5a72892bde7533d867006bff71a3bcf93c0d608c95f4: Status 404 returned error can't find the container with id dc7fa253b5e0125fdadd5a72892bde7533d867006bff71a3bcf93c0d608c95f4 Oct 30 19:30:01.137648 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:01.137621 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6"] Oct 30 19:30:01.185942 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:01.185916 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg"] Oct 30 19:30:01.318956 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:30:01.318860 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod541165d3_33cc_415c_9932_75e6502c70a8.slice/crio-f73b750bfc6ea29b1b1f80bcd51e80eef94ef804c06bbcbab483d2059b867483 WatchSource:0}: Error finding container f73b750bfc6ea29b1b1f80bcd51e80eef94ef804c06bbcbab483d2059b867483: Status 404 returned error can't find the container with id f73b750bfc6ea29b1b1f80bcd51e80eef94ef804c06bbcbab483d2059b867483 Oct 30 19:30:01.418716 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:30:01.418673 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f6ffce_2030_4be2_bf0a_cf4a56dec691.slice/crio-7b931927ef388fdb3f2ff08c02c48049cf9cef7bf6a7c23cb6f95367bad18c92 WatchSource:0}: Error finding container 7b931927ef388fdb3f2ff08c02c48049cf9cef7bf6a7c23cb6f95367bad18c92: Status 404 returned error can't find the container with id 7b931927ef388fdb3f2ff08c02c48049cf9cef7bf6a7c23cb6f95367bad18c92 Oct 30 19:30:01.469885 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:01.469844 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9" event={"ID":"6f45f6f0-a99e-4381-a553-270bdb2f7621","Type":"ContainerStarted","Data":"dc7fa253b5e0125fdadd5a72892bde7533d867006bff71a3bcf93c0d608c95f4"} Oct 30 19:30:01.471518 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:01.471471 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt" event={"ID":"10065963-ac42-42d0-902e-3ec7e87ebd06","Type":"ContainerStarted","Data":"4bbb8506aed02daf18001c76d1f736ea95cb43b2b89e3f29712890e2de038637"} Oct 30 19:30:01.472842 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:01.472813 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w" event={"ID":"0a549c34-f860-401d-a5e6-9c45f015cb54","Type":"ContainerStarted","Data":"75cb69c01ff29b22547af0a8fc9bdc96dfbb3316d2d29911adfb23447ba40eac"} Oct 30 19:30:01.474363 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:01.474333 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6" event={"ID":"541165d3-33cc-415c-9932-75e6502c70a8","Type":"ContainerStarted","Data":"f73b750bfc6ea29b1b1f80bcd51e80eef94ef804c06bbcbab483d2059b867483"} Oct 30 19:30:01.475885 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:01.475854 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7" event={"ID":"5972ca80-ec09-4b6a-be76-847b415f88fb","Type":"ContainerStarted","Data":"0f3c7961e462fafc7891f288db6274a4a19c2b7be3be8a3176182ff9d911be52"} Oct 30 19:30:01.477357 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:01.477321 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l" event={"ID":"747b02b0-b487-4542-9174-862708674b4c","Type":"ContainerStarted","Data":"3e087372270407edf343498ef145c5589863e08bb4b9b178d4087dae73bccb4e"} Oct 30 19:30:01.478728 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:01.478697 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg" event={"ID":"65f6ffce-2030-4be2-bf0a-cf4a56dec691","Type":"ContainerStarted","Data":"7b931927ef388fdb3f2ff08c02c48049cf9cef7bf6a7c23cb6f95367bad18c92"} Oct 30 19:30:01.480064 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:01.480031 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9" event={"ID":"15436c31-541b-4441-b020-dc795e1387aa","Type":"ContainerStarted","Data":"30eb3363637d8dde29ef6fd1598c9d94edfe5f66fb29e2091dac7889b4a41c3b"} Oct 30 19:30:01.481345 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:01.481321 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb" event={"ID":"8d3ce71a-6c7f-4aca-829e-10b51893d417","Type":"ContainerStarted","Data":"e514fb251de854c2d073b36e3cdd152f1368a57793dcb925e1fe6aae008dc5b1"} Oct 30 19:30:02.491001 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:02.490950 4531 generic.go:356] "Generic (PLEG): container finished" podID="8d3ce71a-6c7f-4aca-829e-10b51893d417" containerID="73905438ac156d8aad55606eb2753f6fc83578b6d78b5187a716ad143c650a08" exitCode=60 Oct 30 19:30:02.491500 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:02.491035 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb" event={"ID":"8d3ce71a-6c7f-4aca-829e-10b51893d417","Type":"ContainerDied","Data":"73905438ac156d8aad55606eb2753f6fc83578b6d78b5187a716ad143c650a08"} Oct 30 19:30:02.491500 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:02.491473 4531 scope.go:122] "RemoveContainer" containerID="73905438ac156d8aad55606eb2753f6fc83578b6d78b5187a716ad143c650a08" Oct 30 19:30:02.493102 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:02.493071 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9" event={"ID":"6f45f6f0-a99e-4381-a553-270bdb2f7621","Type":"ContainerStarted","Data":"ba5a79b532da35feb3d25c0067dea2e4d9b62cac388d6fcf45b101dc74cf5a34"} Oct 30 19:30:02.494930 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:02.494903 4531 generic.go:356] "Generic (PLEG): container finished" podID="10065963-ac42-42d0-902e-3ec7e87ebd06" containerID="d6ffd5b28c9508c793587b046e390052f41d646156638cfa2fc79aa6572ba57d" exitCode=60 Oct 30 19:30:02.495063 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:02.495036 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt" event={"ID":"10065963-ac42-42d0-902e-3ec7e87ebd06","Type":"ContainerDied","Data":"d6ffd5b28c9508c793587b046e390052f41d646156638cfa2fc79aa6572ba57d"} Oct 30 19:30:02.495361 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:02.495345 4531 scope.go:122] "RemoveContainer" containerID="d6ffd5b28c9508c793587b046e390052f41d646156638cfa2fc79aa6572ba57d" Oct 30 19:30:02.497060 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:02.497031 4531 generic.go:356] "Generic (PLEG): container finished" podID="0a549c34-f860-401d-a5e6-9c45f015cb54" containerID="13cf746e48e94e586535690ceb5eaaffb4702201b73d4a2c1f67e23f6461d00b" exitCode=60 Oct 30 19:30:02.497204 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:02.497113 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w" event={"ID":"0a549c34-f860-401d-a5e6-9c45f015cb54","Type":"ContainerDied","Data":"13cf746e48e94e586535690ceb5eaaffb4702201b73d4a2c1f67e23f6461d00b"} Oct 30 19:30:02.497426 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:02.497410 4531 scope.go:122] "RemoveContainer" containerID="13cf746e48e94e586535690ceb5eaaffb4702201b73d4a2c1f67e23f6461d00b" Oct 30 19:30:02.499199 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:02.499167 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6" event={"ID":"541165d3-33cc-415c-9932-75e6502c70a8","Type":"ContainerStarted","Data":"ccd2fb6d86b3011355e127d5babfc6556ab08f16778e71fbd552ed7fb681c1bc"} Oct 30 19:30:02.501288 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:02.501242 4531 generic.go:356] "Generic (PLEG): container finished" podID="5972ca80-ec09-4b6a-be76-847b415f88fb" containerID="e4db862e3a56946c10bbc8459f3a8736e789a13340695c1e4ecdc8c0455ab4f7" exitCode=60 Oct 30 19:30:02.501429 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:02.501359 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7" event={"ID":"5972ca80-ec09-4b6a-be76-847b415f88fb","Type":"ContainerDied","Data":"e4db862e3a56946c10bbc8459f3a8736e789a13340695c1e4ecdc8c0455ab4f7"} Oct 30 19:30:02.501764 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:02.501739 4531 scope.go:122] "RemoveContainer" containerID="e4db862e3a56946c10bbc8459f3a8736e789a13340695c1e4ecdc8c0455ab4f7" Oct 30 19:30:02.503357 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:02.503331 4531 generic.go:356] "Generic (PLEG): container finished" podID="747b02b0-b487-4542-9174-862708674b4c" containerID="e6464a5d2c53aba67a7457a070cbba70661df17acbbe19ebf2d1fdf5627844dd" exitCode=60 Oct 30 19:30:02.503543 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:02.503410 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l" event={"ID":"747b02b0-b487-4542-9174-862708674b4c","Type":"ContainerDied","Data":"e6464a5d2c53aba67a7457a070cbba70661df17acbbe19ebf2d1fdf5627844dd"} Oct 30 19:30:02.503917 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:02.503739 4531 scope.go:122] "RemoveContainer" containerID="e6464a5d2c53aba67a7457a070cbba70661df17acbbe19ebf2d1fdf5627844dd" Oct 30 19:30:02.511010 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:02.506270 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg" event={"ID":"65f6ffce-2030-4be2-bf0a-cf4a56dec691","Type":"ContainerStarted","Data":"e79f059cdabd2053e76cd80837b0578cfa6842f427c2bc3bc2caa3cb50fd0b33"} Oct 30 19:30:02.511010 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:02.508758 4531 generic.go:356] "Generic (PLEG): container finished" podID="15436c31-541b-4441-b020-dc795e1387aa" containerID="32b090ee212a761e4ef68ff2d6b650fa6bc69ca01c45c0980f770a6a63cf7a25" exitCode=60 Oct 30 19:30:02.511010 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:02.508811 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9" event={"ID":"15436c31-541b-4441-b020-dc795e1387aa","Type":"ContainerDied","Data":"32b090ee212a761e4ef68ff2d6b650fa6bc69ca01c45c0980f770a6a63cf7a25"} Oct 30 19:30:02.511010 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:02.509176 4531 scope.go:122] "RemoveContainer" containerID="32b090ee212a761e4ef68ff2d6b650fa6bc69ca01c45c0980f770a6a63cf7a25" Oct 30 19:30:02.684680 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:02.684629 4531 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f6ffce_2030_4be2_bf0a_cf4a56dec691.slice/crio-e79f059cdabd2053e76cd80837b0578cfa6842f427c2bc3bc2caa3cb50fd0b33.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:30:02.724563 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:02.724505 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6" podStartSLOduration=2.72448598 podStartE2EDuration="2.72448598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:02.724070054 +0000 UTC m=+3362.751175021" watchObservedRunningTime="2026-10-30 19:30:02.72448598 +0000 UTC m=+3362.751590943" Oct 30 19:30:03.071675 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:03.071558 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9" podStartSLOduration=3.071540306 podStartE2EDuration="3.071540306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:03.068879893 +0000 UTC m=+3363.095984852" watchObservedRunningTime="2026-10-30 19:30:03.071540306 +0000 UTC m=+3363.098645262" Oct 30 19:30:03.186948 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:03.186894 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg" podStartSLOduration=3.186881382 podStartE2EDuration="3.186881382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:03.185340089 +0000 UTC m=+3363.212445055" watchObservedRunningTime="2026-10-30 19:30:03.186881382 +0000 UTC m=+3363.213986340" Oct 30 19:30:03.519526 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:03.519038 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb" event={"ID":"8d3ce71a-6c7f-4aca-829e-10b51893d417","Type":"ContainerStarted","Data":"52726ceec5855295a955d2e05c868e44820f97d3b6769e9779dbe952da25f7e4"} Oct 30 19:30:03.521560 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:03.521533 4531 generic.go:356] "Generic (PLEG): container finished" podID="6f45f6f0-a99e-4381-a553-270bdb2f7621" containerID="ba5a79b532da35feb3d25c0067dea2e4d9b62cac388d6fcf45b101dc74cf5a34" exitCode=60 Oct 30 19:30:03.521701 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:03.521603 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9" event={"ID":"6f45f6f0-a99e-4381-a553-270bdb2f7621","Type":"ContainerDied","Data":"ba5a79b532da35feb3d25c0067dea2e4d9b62cac388d6fcf45b101dc74cf5a34"} Oct 30 19:30:03.521887 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:03.521872 4531 scope.go:122] "RemoveContainer" containerID="ba5a79b532da35feb3d25c0067dea2e4d9b62cac388d6fcf45b101dc74cf5a34" Oct 30 19:30:03.523918 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:03.523894 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt" event={"ID":"10065963-ac42-42d0-902e-3ec7e87ebd06","Type":"ContainerStarted","Data":"c865e8cb697f1fd39b883c456e9b929f7e1fc214e40602497c2f25c185a21caa"} Oct 30 19:30:03.527510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:03.527388 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w" event={"ID":"0a549c34-f860-401d-a5e6-9c45f015cb54","Type":"ContainerStarted","Data":"f1aa2f4bbaf0126a79df40a1435cf09050852379b722fe3f11e98be63309c4b6"} Oct 30 19:30:03.528886 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:03.528860 4531 generic.go:356] "Generic (PLEG): container finished" podID="541165d3-33cc-415c-9932-75e6502c70a8" containerID="ccd2fb6d86b3011355e127d5babfc6556ab08f16778e71fbd552ed7fb681c1bc" exitCode=60 Oct 30 19:30:03.528995 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:03.528937 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6" event={"ID":"541165d3-33cc-415c-9932-75e6502c70a8","Type":"ContainerDied","Data":"ccd2fb6d86b3011355e127d5babfc6556ab08f16778e71fbd552ed7fb681c1bc"} Oct 30 19:30:03.529107 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:03.529092 4531 scope.go:122] "RemoveContainer" containerID="ccd2fb6d86b3011355e127d5babfc6556ab08f16778e71fbd552ed7fb681c1bc" Oct 30 19:30:03.530835 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:03.530816 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7" event={"ID":"5972ca80-ec09-4b6a-be76-847b415f88fb","Type":"ContainerStarted","Data":"a7d76a80c36aebaa5e611959af85f5a13c8033197f6bc28b9f09627a6f923ad7"} Oct 30 19:30:03.533377 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:03.533356 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l" event={"ID":"747b02b0-b487-4542-9174-862708674b4c","Type":"ContainerStarted","Data":"db8876aad79951b7b0cabe3e2728335c43b0af1f3e6f08ecdcfb6d480f3ded05"} Oct 30 19:30:03.535624 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:03.535505 4531 generic.go:356] "Generic (PLEG): container finished" podID="65f6ffce-2030-4be2-bf0a-cf4a56dec691" containerID="e79f059cdabd2053e76cd80837b0578cfa6842f427c2bc3bc2caa3cb50fd0b33" exitCode=60 Oct 30 19:30:03.535624 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:03.535572 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg" event={"ID":"65f6ffce-2030-4be2-bf0a-cf4a56dec691","Type":"ContainerDied","Data":"e79f059cdabd2053e76cd80837b0578cfa6842f427c2bc3bc2caa3cb50fd0b33"} Oct 30 19:30:03.535760 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:03.535745 4531 scope.go:122] "RemoveContainer" containerID="e79f059cdabd2053e76cd80837b0578cfa6842f427c2bc3bc2caa3cb50fd0b33" Oct 30 19:30:03.537700 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:03.537677 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9" event={"ID":"15436c31-541b-4441-b020-dc795e1387aa","Type":"ContainerStarted","Data":"bf0950286f5261630a86e4f9ba95e6efe1388a423c6d51f257e817f20a9961a0"} Oct 30 19:30:03.649397 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:03.649340 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt" podStartSLOduration=3.649325443 podStartE2EDuration="3.649325443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:03.646777288 +0000 UTC m=+3363.673882250" watchObservedRunningTime="2026-10-30 19:30:03.649325443 +0000 UTC m=+3363.676430401" Oct 30 19:30:03.764802 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:03.764744 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w" podStartSLOduration=3.764727493 podStartE2EDuration="3.764727493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:03.76208039 +0000 UTC m=+3363.789185348" watchObservedRunningTime="2026-10-30 19:30:03.764727493 +0000 UTC m=+3363.791832452" Oct 30 19:30:03.995465 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:03.995400 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7" podStartSLOduration=3.995380134 podStartE2EDuration="3.995380134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:03.993457436 +0000 UTC m=+3364.020562396" watchObservedRunningTime="2026-10-30 19:30:03.995380134 +0000 UTC m=+3364.022485093" Oct 30 19:30:04.111349 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.111280 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb" podStartSLOduration=4.111262603 podStartE2EDuration="4.111262603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:04.108384088 +0000 UTC m=+3364.135489050" watchObservedRunningTime="2026-10-30 19:30:04.111262603 +0000 UTC m=+3364.138367561" Oct 30 19:30:04.459742 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.459677 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9" podStartSLOduration=4.459659473 podStartE2EDuration="4.459659473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:04.457032885 +0000 UTC m=+3364.484137844" watchObservedRunningTime="2026-10-30 19:30:04.459659473 +0000 UTC m=+3364.486764506" Oct 30 19:30:04.555476 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.555426 4531 generic.go:356] "Generic (PLEG): container finished" podID="10065963-ac42-42d0-902e-3ec7e87ebd06" containerID="c865e8cb697f1fd39b883c456e9b929f7e1fc214e40602497c2f25c185a21caa" exitCode=60 Oct 30 19:30:04.555987 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.555509 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt" event={"ID":"10065963-ac42-42d0-902e-3ec7e87ebd06","Type":"ContainerDied","Data":"c865e8cb697f1fd39b883c456e9b929f7e1fc214e40602497c2f25c185a21caa"} Oct 30 19:30:04.555987 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.555611 4531 scope.go:122] "RemoveContainer" containerID="d6ffd5b28c9508c793587b046e390052f41d646156638cfa2fc79aa6572ba57d" Oct 30 19:30:04.555987 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.555789 4531 scope.go:122] "RemoveContainer" containerID="c865e8cb697f1fd39b883c456e9b929f7e1fc214e40602497c2f25c185a21caa" Oct 30 19:30:04.556191 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:04.556105 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt_odh-ai-gateway-infra(10065963-ac42-42d0-902e-3ec7e87ebd06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt" podUID="10065963-ac42-42d0-902e-3ec7e87ebd06" Oct 30 19:30:04.558217 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.558192 4531 generic.go:356] "Generic (PLEG): container finished" podID="0a549c34-f860-401d-a5e6-9c45f015cb54" containerID="f1aa2f4bbaf0126a79df40a1435cf09050852379b722fe3f11e98be63309c4b6" exitCode=60 Oct 30 19:30:04.558356 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.558277 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w" event={"ID":"0a549c34-f860-401d-a5e6-9c45f015cb54","Type":"ContainerDied","Data":"f1aa2f4bbaf0126a79df40a1435cf09050852379b722fe3f11e98be63309c4b6"} Oct 30 19:30:04.558641 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.558624 4531 scope.go:122] "RemoveContainer" containerID="f1aa2f4bbaf0126a79df40a1435cf09050852379b722fe3f11e98be63309c4b6" Oct 30 19:30:04.558954 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:04.558934 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w_odh-ai-gateway-infra(0a549c34-f860-401d-a5e6-9c45f015cb54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w" podUID="0a549c34-f860-401d-a5e6-9c45f015cb54" Oct 30 19:30:04.560648 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.560616 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6" event={"ID":"541165d3-33cc-415c-9932-75e6502c70a8","Type":"ContainerStarted","Data":"d1347626378fc58158f46e3726a6ac554efd2110cefe5ee0d44afbd87b232b80"} Oct 30 19:30:04.562482 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.562456 4531 generic.go:356] "Generic (PLEG): container finished" podID="5972ca80-ec09-4b6a-be76-847b415f88fb" containerID="a7d76a80c36aebaa5e611959af85f5a13c8033197f6bc28b9f09627a6f923ad7" exitCode=60 Oct 30 19:30:04.562623 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.562542 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7" event={"ID":"5972ca80-ec09-4b6a-be76-847b415f88fb","Type":"ContainerDied","Data":"a7d76a80c36aebaa5e611959af85f5a13c8033197f6bc28b9f09627a6f923ad7"} Oct 30 19:30:04.562892 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.562876 4531 scope.go:122] "RemoveContainer" containerID="a7d76a80c36aebaa5e611959af85f5a13c8033197f6bc28b9f09627a6f923ad7" Oct 30 19:30:04.563216 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:04.563194 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7_odh-ai-gateway-infra(5972ca80-ec09-4b6a-be76-847b415f88fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7" podUID="5972ca80-ec09-4b6a-be76-847b415f88fb" Oct 30 19:30:04.564612 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.564588 4531 generic.go:356] "Generic (PLEG): container finished" podID="747b02b0-b487-4542-9174-862708674b4c" containerID="db8876aad79951b7b0cabe3e2728335c43b0af1f3e6f08ecdcfb6d480f3ded05" exitCode=60 Oct 30 19:30:04.564710 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.564674 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l" event={"ID":"747b02b0-b487-4542-9174-862708674b4c","Type":"ContainerDied","Data":"db8876aad79951b7b0cabe3e2728335c43b0af1f3e6f08ecdcfb6d480f3ded05"} Oct 30 19:30:04.564990 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.564974 4531 scope.go:122] "RemoveContainer" containerID="db8876aad79951b7b0cabe3e2728335c43b0af1f3e6f08ecdcfb6d480f3ded05" Oct 30 19:30:04.565339 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:04.565317 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l_odh-ai-gateway-infra(747b02b0-b487-4542-9174-862708674b4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l" podUID="747b02b0-b487-4542-9174-862708674b4c" Oct 30 19:30:04.566838 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.566807 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg" event={"ID":"65f6ffce-2030-4be2-bf0a-cf4a56dec691","Type":"ContainerStarted","Data":"658ef9717fb3e7f000b5c8a5f558b11c207b108fd8f5ee8a9f0488072990db0e"} Oct 30 19:30:04.568825 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.568798 4531 generic.go:356] "Generic (PLEG): container finished" podID="15436c31-541b-4441-b020-dc795e1387aa" containerID="bf0950286f5261630a86e4f9ba95e6efe1388a423c6d51f257e817f20a9961a0" exitCode=60 Oct 30 19:30:04.568919 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.568874 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9" event={"ID":"15436c31-541b-4441-b020-dc795e1387aa","Type":"ContainerDied","Data":"bf0950286f5261630a86e4f9ba95e6efe1388a423c6d51f257e817f20a9961a0"} Oct 30 19:30:04.569121 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.569104 4531 scope.go:122] "RemoveContainer" containerID="bf0950286f5261630a86e4f9ba95e6efe1388a423c6d51f257e817f20a9961a0" Oct 30 19:30:04.569452 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:04.569427 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9_odh-ai-gateway-infra(15436c31-541b-4441-b020-dc795e1387aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9" podUID="15436c31-541b-4441-b020-dc795e1387aa" Oct 30 19:30:04.570455 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.570437 4531 scope.go:122] "RemoveContainer" containerID="13cf746e48e94e586535690ceb5eaaffb4702201b73d4a2c1f67e23f6461d00b" Oct 30 19:30:04.571012 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.570991 4531 generic.go:356] "Generic (PLEG): container finished" podID="8d3ce71a-6c7f-4aca-829e-10b51893d417" containerID="52726ceec5855295a955d2e05c868e44820f97d3b6769e9779dbe952da25f7e4" exitCode=60 Oct 30 19:30:04.571122 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.571100 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb" event={"ID":"8d3ce71a-6c7f-4aca-829e-10b51893d417","Type":"ContainerDied","Data":"52726ceec5855295a955d2e05c868e44820f97d3b6769e9779dbe952da25f7e4"} Oct 30 19:30:04.571443 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.571429 4531 scope.go:122] "RemoveContainer" containerID="52726ceec5855295a955d2e05c868e44820f97d3b6769e9779dbe952da25f7e4" Oct 30 19:30:04.571813 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:04.571791 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb_odh-ai-gateway-infra(8d3ce71a-6c7f-4aca-829e-10b51893d417)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb" podUID="8d3ce71a-6c7f-4aca-829e-10b51893d417" Oct 30 19:30:04.573298 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.573263 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9" event={"ID":"6f45f6f0-a99e-4381-a553-270bdb2f7621","Type":"ContainerStarted","Data":"e366bb743bd0056ae25ad7baf95bb2cb428a20c6033b0e83a3a6f9c945e9474e"} Oct 30 19:30:04.579148 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.579063 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l" podStartSLOduration=4.579041114 podStartE2EDuration="4.579041114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:04.576459388 +0000 UTC m=+3364.603564347" watchObservedRunningTime="2026-10-30 19:30:04.579041114 +0000 UTC m=+3364.606146102" Oct 30 19:30:04.586165 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.586144 4531 scope.go:122] "RemoveContainer" containerID="e4db862e3a56946c10bbc8459f3a8736e789a13340695c1e4ecdc8c0455ab4f7" Oct 30 19:30:04.600851 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.600821 4531 scope.go:122] "RemoveContainer" containerID="e6464a5d2c53aba67a7457a070cbba70661df17acbbe19ebf2d1fdf5627844dd" Oct 30 19:30:04.615181 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.615156 4531 scope.go:122] "RemoveContainer" containerID="32b090ee212a761e4ef68ff2d6b650fa6bc69ca01c45c0980f770a6a63cf7a25" Oct 30 19:30:04.697619 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:04.697593 4531 scope.go:122] "RemoveContainer" containerID="73905438ac156d8aad55606eb2753f6fc83578b6d78b5187a716ad143c650a08" Oct 30 19:30:05.582311 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:05.582280 4531 scope.go:122] "RemoveContainer" containerID="f1aa2f4bbaf0126a79df40a1435cf09050852379b722fe3f11e98be63309c4b6" Oct 30 19:30:05.582808 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:05.582604 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w_odh-ai-gateway-infra(0a549c34-f860-401d-a5e6-9c45f015cb54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w" podUID="0a549c34-f860-401d-a5e6-9c45f015cb54" Oct 30 19:30:05.584253 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:05.584219 4531 generic.go:356] "Generic (PLEG): container finished" podID="541165d3-33cc-415c-9932-75e6502c70a8" containerID="d1347626378fc58158f46e3726a6ac554efd2110cefe5ee0d44afbd87b232b80" exitCode=60 Oct 30 19:30:05.584386 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:05.584286 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6" event={"ID":"541165d3-33cc-415c-9932-75e6502c70a8","Type":"ContainerDied","Data":"d1347626378fc58158f46e3726a6ac554efd2110cefe5ee0d44afbd87b232b80"} Oct 30 19:30:05.584386 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:05.584348 4531 scope.go:122] "RemoveContainer" containerID="ccd2fb6d86b3011355e127d5babfc6556ab08f16778e71fbd552ed7fb681c1bc" Oct 30 19:30:05.584646 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:05.584629 4531 scope.go:122] "RemoveContainer" containerID="d1347626378fc58158f46e3726a6ac554efd2110cefe5ee0d44afbd87b232b80" Oct 30 19:30:05.584988 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:05.584966 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6_odh-ai-gateway-infra(541165d3-33cc-415c-9932-75e6502c70a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6" podUID="541165d3-33cc-415c-9932-75e6502c70a8" Oct 30 19:30:05.586507 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:05.586486 4531 scope.go:122] "RemoveContainer" containerID="a7d76a80c36aebaa5e611959af85f5a13c8033197f6bc28b9f09627a6f923ad7" Oct 30 19:30:05.586801 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:05.586773 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7_odh-ai-gateway-infra(5972ca80-ec09-4b6a-be76-847b415f88fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7" podUID="5972ca80-ec09-4b6a-be76-847b415f88fb" Oct 30 19:30:05.588345 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:05.588319 4531 scope.go:122] "RemoveContainer" containerID="db8876aad79951b7b0cabe3e2728335c43b0af1f3e6f08ecdcfb6d480f3ded05" Oct 30 19:30:05.588669 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:05.588644 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l_odh-ai-gateway-infra(747b02b0-b487-4542-9174-862708674b4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l" podUID="747b02b0-b487-4542-9174-862708674b4c" Oct 30 19:30:05.590293 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:05.590250 4531 generic.go:356] "Generic (PLEG): container finished" podID="65f6ffce-2030-4be2-bf0a-cf4a56dec691" containerID="658ef9717fb3e7f000b5c8a5f558b11c207b108fd8f5ee8a9f0488072990db0e" exitCode=60 Oct 30 19:30:05.590415 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:05.590370 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg" event={"ID":"65f6ffce-2030-4be2-bf0a-cf4a56dec691","Type":"ContainerDied","Data":"658ef9717fb3e7f000b5c8a5f558b11c207b108fd8f5ee8a9f0488072990db0e"} Oct 30 19:30:05.590712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:05.590690 4531 scope.go:122] "RemoveContainer" containerID="658ef9717fb3e7f000b5c8a5f558b11c207b108fd8f5ee8a9f0488072990db0e" Oct 30 19:30:05.591050 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:05.591027 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg_odh-ai-gateway-infra(65f6ffce-2030-4be2-bf0a-cf4a56dec691)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg" podUID="65f6ffce-2030-4be2-bf0a-cf4a56dec691" Oct 30 19:30:05.592829 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:05.592805 4531 scope.go:122] "RemoveContainer" containerID="bf0950286f5261630a86e4f9ba95e6efe1388a423c6d51f257e817f20a9961a0" Oct 30 19:30:05.593342 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:05.593169 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9_odh-ai-gateway-infra(15436c31-541b-4441-b020-dc795e1387aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9" podUID="15436c31-541b-4441-b020-dc795e1387aa" Oct 30 19:30:05.594940 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:05.594917 4531 scope.go:122] "RemoveContainer" containerID="52726ceec5855295a955d2e05c868e44820f97d3b6769e9779dbe952da25f7e4" Oct 30 19:30:05.595281 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:05.595245 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb_odh-ai-gateway-infra(8d3ce71a-6c7f-4aca-829e-10b51893d417)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb" podUID="8d3ce71a-6c7f-4aca-829e-10b51893d417" Oct 30 19:30:05.596973 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:05.596930 4531 generic.go:356] "Generic (PLEG): container finished" podID="6f45f6f0-a99e-4381-a553-270bdb2f7621" containerID="e366bb743bd0056ae25ad7baf95bb2cb428a20c6033b0e83a3a6f9c945e9474e" exitCode=60 Oct 30 19:30:05.597119 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:05.597060 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9" event={"ID":"6f45f6f0-a99e-4381-a553-270bdb2f7621","Type":"ContainerDied","Data":"e366bb743bd0056ae25ad7baf95bb2cb428a20c6033b0e83a3a6f9c945e9474e"} Oct 30 19:30:05.597370 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:05.597355 4531 scope.go:122] "RemoveContainer" containerID="e366bb743bd0056ae25ad7baf95bb2cb428a20c6033b0e83a3a6f9c945e9474e" Oct 30 19:30:05.597754 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:05.597730 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9_odh-ai-gateway-infra(6f45f6f0-a99e-4381-a553-270bdb2f7621)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9" podUID="6f45f6f0-a99e-4381-a553-270bdb2f7621" Oct 30 19:30:05.600272 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:05.600242 4531 scope.go:122] "RemoveContainer" containerID="c865e8cb697f1fd39b883c456e9b929f7e1fc214e40602497c2f25c185a21caa" Oct 30 19:30:05.600833 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:05.600797 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt_odh-ai-gateway-infra(10065963-ac42-42d0-902e-3ec7e87ebd06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt" podUID="10065963-ac42-42d0-902e-3ec7e87ebd06" Oct 30 19:30:05.601569 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:05.601549 4531 scope.go:122] "RemoveContainer" containerID="e79f059cdabd2053e76cd80837b0578cfa6842f427c2bc3bc2caa3cb50fd0b33" Oct 30 19:30:05.617029 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:05.617004 4531 scope.go:122] "RemoveContainer" containerID="ba5a79b532da35feb3d25c0067dea2e4d9b62cac388d6fcf45b101dc74cf5a34" Oct 30 19:30:06.610269 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:06.610237 4531 scope.go:122] "RemoveContainer" containerID="e366bb743bd0056ae25ad7baf95bb2cb428a20c6033b0e83a3a6f9c945e9474e" Oct 30 19:30:06.610754 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:06.610556 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9_odh-ai-gateway-infra(6f45f6f0-a99e-4381-a553-270bdb2f7621)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9" podUID="6f45f6f0-a99e-4381-a553-270bdb2f7621" Oct 30 19:30:06.612081 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:06.612057 4531 scope.go:122] "RemoveContainer" containerID="d1347626378fc58158f46e3726a6ac554efd2110cefe5ee0d44afbd87b232b80" Oct 30 19:30:06.612413 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:06.612382 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6_odh-ai-gateway-infra(541165d3-33cc-415c-9932-75e6502c70a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6" podUID="541165d3-33cc-415c-9932-75e6502c70a8" Oct 30 19:30:06.613776 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:06.613754 4531 scope.go:122] "RemoveContainer" containerID="658ef9717fb3e7f000b5c8a5f558b11c207b108fd8f5ee8a9f0488072990db0e" Oct 30 19:30:06.614038 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:06.614017 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg_odh-ai-gateway-infra(65f6ffce-2030-4be2-bf0a-cf4a56dec691)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg" podUID="65f6ffce-2030-4be2-bf0a-cf4a56dec691" Oct 30 19:30:15.091904 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:15.091819 4531 scope.go:122] "RemoveContainer" containerID="52726ceec5855295a955d2e05c868e44820f97d3b6769e9779dbe952da25f7e4" Oct 30 19:30:15.376560 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:15.376462 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7106-5c59ab"] Oct 30 19:30:15.381682 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:15.381643 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7106-5c59ab" Oct 30 19:30:15.435631 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:15.435584 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7q8sq\"" Oct 30 19:30:15.439508 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:15.439468 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7106-5c59ab"] Oct 30 19:30:15.454120 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:15.454079 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgl2l\" (UniqueName: \"kubernetes.io/projected/824fa435-1ea3-424e-9c73-5bbad49280b9-kube-api-access-sgl2l\") pod \"test-curl-7106-5c59ab\" (UID: \"824fa435-1ea3-424e-9c73-5bbad49280b9\") " pod="opendatahub/test-curl-7106-5c59ab" Oct 30 19:30:15.555633 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:15.555585 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgl2l\" (UniqueName: \"kubernetes.io/projected/824fa435-1ea3-424e-9c73-5bbad49280b9-kube-api-access-sgl2l\") pod \"test-curl-7106-5c59ab\" (UID: \"824fa435-1ea3-424e-9c73-5bbad49280b9\") " pod="opendatahub/test-curl-7106-5c59ab" Oct 30 19:30:15.617672 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:15.617631 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgl2l\" (UniqueName: \"kubernetes.io/projected/824fa435-1ea3-424e-9c73-5bbad49280b9-kube-api-access-sgl2l\") pod \"test-curl-7106-5c59ab\" (UID: \"824fa435-1ea3-424e-9c73-5bbad49280b9\") " pod="opendatahub/test-curl-7106-5c59ab" Oct 30 19:30:15.673298 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:15.673211 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb" event={"ID":"8d3ce71a-6c7f-4aca-829e-10b51893d417","Type":"ContainerStarted","Data":"4439fb353230ed310cc1b12c5a39c0af46a9545a9a3bff7f527e59787c5b418f"} Oct 30 19:30:15.695043 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:15.695006 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7106-5c59ab" Oct 30 19:30:16.078802 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:16.078773 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7106-5c59ab"] Oct 30 19:30:16.090289 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:30:16.090261 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod824fa435_1ea3_424e_9c73_5bbad49280b9.slice/crio-35aa412e7ee132927baa527b8a2dd1b6c1730cababb8e98d6df2369ecc68cc65 WatchSource:0}: Error finding container 35aa412e7ee132927baa527b8a2dd1b6c1730cababb8e98d6df2369ecc68cc65: Status 404 returned error can't find the container with id 35aa412e7ee132927baa527b8a2dd1b6c1730cababb8e98d6df2369ecc68cc65 Oct 30 19:30:16.091621 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:16.091599 4531 scope.go:122] "RemoveContainer" containerID="f1aa2f4bbaf0126a79df40a1435cf09050852379b722fe3f11e98be63309c4b6" Oct 30 19:30:16.091828 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:16.091797 4531 scope.go:122] "RemoveContainer" containerID="db8876aad79951b7b0cabe3e2728335c43b0af1f3e6f08ecdcfb6d480f3ded05" Oct 30 19:30:16.091971 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:16.091916 4531 scope.go:122] "RemoveContainer" containerID="bf0950286f5261630a86e4f9ba95e6efe1388a423c6d51f257e817f20a9961a0" Oct 30 19:30:16.092393 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:16.092123 4531 scope.go:122] "RemoveContainer" containerID="658ef9717fb3e7f000b5c8a5f558b11c207b108fd8f5ee8a9f0488072990db0e" Oct 30 19:30:16.092393 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:16.092233 4531 scope.go:122] "RemoveContainer" containerID="a7d76a80c36aebaa5e611959af85f5a13c8033197f6bc28b9f09627a6f923ad7" Oct 30 19:30:16.681880 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:16.681844 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w" event={"ID":"0a549c34-f860-401d-a5e6-9c45f015cb54","Type":"ContainerStarted","Data":"023e434c8d6a2b67ef18118dbb4f5933767f3f5ff1cd9caea31066b6cf93df90"} Oct 30 19:30:16.683828 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:16.683794 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7" event={"ID":"5972ca80-ec09-4b6a-be76-847b415f88fb","Type":"ContainerStarted","Data":"b43ed4da879787967cb8efc667923330ba0d3c56f76cee5a79522a017dc270bf"} Oct 30 19:30:16.685748 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:16.685721 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l" event={"ID":"747b02b0-b487-4542-9174-862708674b4c","Type":"ContainerStarted","Data":"ccc208a585c45bfd5ab152e51f24c51164d7e62a845e161427534684a44ee224"} Oct 30 19:30:16.687672 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:16.687648 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg" event={"ID":"65f6ffce-2030-4be2-bf0a-cf4a56dec691","Type":"ContainerStarted","Data":"5d2b150e073958013817f81141baf8df6a226e89e1f30d2de92d76b0ec6771f0"} Oct 30 19:30:16.689395 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:16.689367 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7106-5c59ab" event={"ID":"824fa435-1ea3-424e-9c73-5bbad49280b9","Type":"ContainerStarted","Data":"901ca5943ade131beef9ef604cedb9826da1b25888b5fddcd79531437c4ac36d"} Oct 30 19:30:16.689533 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:16.689406 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7106-5c59ab" event={"ID":"824fa435-1ea3-424e-9c73-5bbad49280b9","Type":"ContainerStarted","Data":"35aa412e7ee132927baa527b8a2dd1b6c1730cababb8e98d6df2369ecc68cc65"} Oct 30 19:30:16.691664 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:16.691585 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9" event={"ID":"15436c31-541b-4441-b020-dc795e1387aa","Type":"ContainerStarted","Data":"290bdf054c8d9babc109677a1b3e6b7f3d355ba95a5af5ae11fa5b5c7312d2bb"} Oct 30 19:30:16.693971 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:16.693943 4531 generic.go:356] "Generic (PLEG): container finished" podID="8d3ce71a-6c7f-4aca-829e-10b51893d417" containerID="4439fb353230ed310cc1b12c5a39c0af46a9545a9a3bff7f527e59787c5b418f" exitCode=60 Oct 30 19:30:16.694101 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:16.693977 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb" event={"ID":"8d3ce71a-6c7f-4aca-829e-10b51893d417","Type":"ContainerDied","Data":"4439fb353230ed310cc1b12c5a39c0af46a9545a9a3bff7f527e59787c5b418f"} Oct 30 19:30:16.694101 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:16.694044 4531 scope.go:122] "RemoveContainer" containerID="52726ceec5855295a955d2e05c868e44820f97d3b6769e9779dbe952da25f7e4" Oct 30 19:30:16.694429 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:16.694413 4531 scope.go:122] "RemoveContainer" containerID="4439fb353230ed310cc1b12c5a39c0af46a9545a9a3bff7f527e59787c5b418f" Oct 30 19:30:16.694699 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:16.694679 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb_odh-ai-gateway-infra(8d3ce71a-6c7f-4aca-829e-10b51893d417)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb" podUID="8d3ce71a-6c7f-4aca-829e-10b51893d417" Oct 30 19:30:16.803419 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:16.803373 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb"] Oct 30 19:30:17.091717 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.091687 4531 scope.go:122] "RemoveContainer" containerID="e366bb743bd0056ae25ad7baf95bb2cb428a20c6033b0e83a3a6f9c945e9474e" Oct 30 19:30:17.492833 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.492762 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7106-5c59ab" podStartSLOduration=2.492740422 podStartE2EDuration="2.492740422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:17.489845579 +0000 UTC m=+3377.516950535" watchObservedRunningTime="2026-10-30 19:30:17.492740422 +0000 UTC m=+3377.519845382" Oct 30 19:30:17.704660 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.704564 4531 generic.go:356] "Generic (PLEG): container finished" podID="0a549c34-f860-401d-a5e6-9c45f015cb54" containerID="023e434c8d6a2b67ef18118dbb4f5933767f3f5ff1cd9caea31066b6cf93df90" exitCode=60 Oct 30 19:30:17.704660 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.704634 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w" event={"ID":"0a549c34-f860-401d-a5e6-9c45f015cb54","Type":"ContainerDied","Data":"023e434c8d6a2b67ef18118dbb4f5933767f3f5ff1cd9caea31066b6cf93df90"} Oct 30 19:30:17.704897 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.704700 4531 scope.go:122] "RemoveContainer" containerID="f1aa2f4bbaf0126a79df40a1435cf09050852379b722fe3f11e98be63309c4b6" Oct 30 19:30:17.705106 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.705088 4531 scope.go:122] "RemoveContainer" containerID="023e434c8d6a2b67ef18118dbb4f5933767f3f5ff1cd9caea31066b6cf93df90" Oct 30 19:30:17.705432 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:17.705406 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w_odh-ai-gateway-infra(0a549c34-f860-401d-a5e6-9c45f015cb54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w" podUID="0a549c34-f860-401d-a5e6-9c45f015cb54" Oct 30 19:30:17.707256 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.707233 4531 generic.go:356] "Generic (PLEG): container finished" podID="5972ca80-ec09-4b6a-be76-847b415f88fb" containerID="b43ed4da879787967cb8efc667923330ba0d3c56f76cee5a79522a017dc270bf" exitCode=60 Oct 30 19:30:17.707375 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.707308 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7" event={"ID":"5972ca80-ec09-4b6a-be76-847b415f88fb","Type":"ContainerDied","Data":"b43ed4da879787967cb8efc667923330ba0d3c56f76cee5a79522a017dc270bf"} Oct 30 19:30:17.707699 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.707675 4531 scope.go:122] "RemoveContainer" containerID="b43ed4da879787967cb8efc667923330ba0d3c56f76cee5a79522a017dc270bf" Oct 30 19:30:17.707996 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:17.707975 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7_odh-ai-gateway-infra(5972ca80-ec09-4b6a-be76-847b415f88fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7" podUID="5972ca80-ec09-4b6a-be76-847b415f88fb" Oct 30 19:30:17.709464 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.709429 4531 generic.go:356] "Generic (PLEG): container finished" podID="747b02b0-b487-4542-9174-862708674b4c" containerID="ccc208a585c45bfd5ab152e51f24c51164d7e62a845e161427534684a44ee224" exitCode=60 Oct 30 19:30:17.709573 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.709510 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l" event={"ID":"747b02b0-b487-4542-9174-862708674b4c","Type":"ContainerDied","Data":"ccc208a585c45bfd5ab152e51f24c51164d7e62a845e161427534684a44ee224"} Oct 30 19:30:17.709942 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.709925 4531 scope.go:122] "RemoveContainer" containerID="ccc208a585c45bfd5ab152e51f24c51164d7e62a845e161427534684a44ee224" Oct 30 19:30:17.710251 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:17.710226 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l_odh-ai-gateway-infra(747b02b0-b487-4542-9174-862708674b4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l" podUID="747b02b0-b487-4542-9174-862708674b4c" Oct 30 19:30:17.711745 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.711717 4531 generic.go:356] "Generic (PLEG): container finished" podID="65f6ffce-2030-4be2-bf0a-cf4a56dec691" containerID="5d2b150e073958013817f81141baf8df6a226e89e1f30d2de92d76b0ec6771f0" exitCode=60 Oct 30 19:30:17.711874 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.711802 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg" event={"ID":"65f6ffce-2030-4be2-bf0a-cf4a56dec691","Type":"ContainerDied","Data":"5d2b150e073958013817f81141baf8df6a226e89e1f30d2de92d76b0ec6771f0"} Oct 30 19:30:17.712255 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.712238 4531 scope.go:122] "RemoveContainer" containerID="5d2b150e073958013817f81141baf8df6a226e89e1f30d2de92d76b0ec6771f0" Oct 30 19:30:17.712574 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:17.712554 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg_odh-ai-gateway-infra(65f6ffce-2030-4be2-bf0a-cf4a56dec691)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg" podUID="65f6ffce-2030-4be2-bf0a-cf4a56dec691" Oct 30 19:30:17.714037 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.714016 4531 generic.go:356] "Generic (PLEG): container finished" podID="15436c31-541b-4441-b020-dc795e1387aa" containerID="290bdf054c8d9babc109677a1b3e6b7f3d355ba95a5af5ae11fa5b5c7312d2bb" exitCode=60 Oct 30 19:30:17.714146 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.714055 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9" event={"ID":"15436c31-541b-4441-b020-dc795e1387aa","Type":"ContainerDied","Data":"290bdf054c8d9babc109677a1b3e6b7f3d355ba95a5af5ae11fa5b5c7312d2bb"} Oct 30 19:30:17.714510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.714487 4531 scope.go:122] "RemoveContainer" containerID="290bdf054c8d9babc109677a1b3e6b7f3d355ba95a5af5ae11fa5b5c7312d2bb" Oct 30 19:30:17.714838 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:17.714816 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9_odh-ai-gateway-infra(15436c31-541b-4441-b020-dc795e1387aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9" podUID="15436c31-541b-4441-b020-dc795e1387aa" Oct 30 19:30:17.718752 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.718676 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9" event={"ID":"6f45f6f0-a99e-4381-a553-270bdb2f7621","Type":"ContainerStarted","Data":"0581696f55496a8da6e323eb1d22aa0d32b1ad58eb676dd39ec191289f671991"} Oct 30 19:30:17.720334 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.720310 4531 scope.go:122] "RemoveContainer" containerID="a7d76a80c36aebaa5e611959af85f5a13c8033197f6bc28b9f09627a6f923ad7" Oct 30 19:30:17.736507 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.736482 4531 scope.go:122] "RemoveContainer" containerID="db8876aad79951b7b0cabe3e2728335c43b0af1f3e6f08ecdcfb6d480f3ded05" Oct 30 19:30:17.765619 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.765590 4531 scope.go:122] "RemoveContainer" containerID="658ef9717fb3e7f000b5c8a5f558b11c207b108fd8f5ee8a9f0488072990db0e" Oct 30 19:30:17.780272 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.780241 4531 scope.go:122] "RemoveContainer" containerID="bf0950286f5261630a86e4f9ba95e6efe1388a423c6d51f257e817f20a9961a0" Oct 30 19:30:17.814492 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.814455 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w"] Oct 30 19:30:17.902873 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.902842 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb" Oct 30 19:30:17.926384 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.926352 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg"] Oct 30 19:30:17.984157 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.984044 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d3ce71a-6c7f-4aca-829e-10b51893d417-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3ce71a-6c7f-4aca-829e-10b51893d417-maas-api-service-ca\") pod \"8d3ce71a-6c7f-4aca-829e-10b51893d417\" (UID: \"8d3ce71a-6c7f-4aca-829e-10b51893d417\") " Oct 30 19:30:17.984321 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.984205 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d3ce71a-6c7f-4aca-829e-10b51893d417-kube-api-access-lfmpg\" (UniqueName: \"kubernetes.io/projected/8d3ce71a-6c7f-4aca-829e-10b51893d417-kube-api-access-lfmpg\") pod \"8d3ce71a-6c7f-4aca-829e-10b51893d417\" (UID: \"8d3ce71a-6c7f-4aca-829e-10b51893d417\") " Oct 30 19:30:17.984511 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.984488 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d3ce71a-6c7f-4aca-829e-10b51893d417-maas-api-service-ca" pod "8d3ce71a-6c7f-4aca-829e-10b51893d417" (UID: "8d3ce71a-6c7f-4aca-829e-10b51893d417"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:17.984634 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.984619 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3ce71a-6c7f-4aca-829e-10b51893d417-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:17.986765 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:17.986734 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d3ce71a-6c7f-4aca-829e-10b51893d417-kube-api-access-lfmpg" pod "8d3ce71a-6c7f-4aca-829e-10b51893d417" (UID: "8d3ce71a-6c7f-4aca-829e-10b51893d417"). InnerVolumeSpecName "kube-api-access-lfmpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:18.039256 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:18.039218 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7"] Oct 30 19:30:18.085580 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:18.085530 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfmpg\" (UniqueName: \"kubernetes.io/projected/8d3ce71a-6c7f-4aca-829e-10b51893d417-kube-api-access-lfmpg\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:18.092376 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:18.092342 4531 scope.go:122] "RemoveContainer" containerID="c865e8cb697f1fd39b883c456e9b929f7e1fc214e40602497c2f25c185a21caa" Oct 30 19:30:18.155208 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:18.155171 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9"] Oct 30 19:30:18.275472 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:18.275432 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l"] Oct 30 19:30:18.727459 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:18.727350 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt" event={"ID":"10065963-ac42-42d0-902e-3ec7e87ebd06","Type":"ContainerStarted","Data":"ec56c817f0e2c0645808a89d2ee176a3feb097cfe4a9e22a7b54b709049f63b2"} Oct 30 19:30:18.746425 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:18.746376 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb" Oct 30 19:30:18.747057 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:18.747017 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb" event={"ID":"8d3ce71a-6c7f-4aca-829e-10b51893d417","Type":"ContainerDied","Data":"e514fb251de854c2d073b36e3cdd152f1368a57793dcb925e1fe6aae008dc5b1"} Oct 30 19:30:18.747220 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:18.747095 4531 scope.go:122] "RemoveContainer" containerID="4439fb353230ed310cc1b12c5a39c0af46a9545a9a3bff7f527e59787c5b418f" Oct 30 19:30:18.755968 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:18.753943 4531 generic.go:356] "Generic (PLEG): container finished" podID="6f45f6f0-a99e-4381-a553-270bdb2f7621" containerID="0581696f55496a8da6e323eb1d22aa0d32b1ad58eb676dd39ec191289f671991" exitCode=60 Oct 30 19:30:18.755968 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:18.753991 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9" event={"ID":"6f45f6f0-a99e-4381-a553-270bdb2f7621","Type":"ContainerDied","Data":"0581696f55496a8da6e323eb1d22aa0d32b1ad58eb676dd39ec191289f671991"} Oct 30 19:30:18.755968 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:18.754415 4531 scope.go:122] "RemoveContainer" containerID="0581696f55496a8da6e323eb1d22aa0d32b1ad58eb676dd39ec191289f671991" Oct 30 19:30:18.755968 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:18.754725 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9_odh-ai-gateway-infra(6f45f6f0-a99e-4381-a553-270bdb2f7621)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9" podUID="6f45f6f0-a99e-4381-a553-270bdb2f7621" Oct 30 19:30:18.768930 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:18.768893 4531 scope.go:122] "RemoveContainer" containerID="e366bb743bd0056ae25ad7baf95bb2cb428a20c6033b0e83a3a6f9c945e9474e" Oct 30 19:30:18.932368 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:18.932267 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l" Oct 30 19:30:18.983501 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:18.980893 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9"] Oct 30 19:30:18.999966 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:18.999774 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7106-5c59ab"] Oct 30 19:30:19.000055 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:18.999986 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7106-5c59ab" podUID="824fa435-1ea3-424e-9c73-5bbad49280b9" containerName="test-curl-7106-5c59ab" containerID="cri-o://901ca5943ade131beef9ef604cedb9826da1b25888b5fddcd79531437c4ac36d" gracePeriod=2 Oct 30 19:30:19.011667 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.011612 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7106-5c59ab"] Oct 30 19:30:19.035892 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.035859 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7" Oct 30 19:30:19.039772 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.039743 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg" Oct 30 19:30:19.045300 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.045272 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w" Oct 30 19:30:19.050032 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.049997 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9" Oct 30 19:30:19.092304 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.092275 4531 scope.go:122] "RemoveContainer" containerID="d1347626378fc58158f46e3726a6ac554efd2110cefe5ee0d44afbd87b232b80" Oct 30 19:30:19.097957 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.097934 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/747b02b0-b487-4542-9174-862708674b4c-kube-api-access-vqcps\" (UniqueName: \"kubernetes.io/projected/747b02b0-b487-4542-9174-862708674b4c-kube-api-access-vqcps\") pod \"747b02b0-b487-4542-9174-862708674b4c\" (UID: \"747b02b0-b487-4542-9174-862708674b4c\") " Oct 30 19:30:19.098057 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.098027 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/747b02b0-b487-4542-9174-862708674b4c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/747b02b0-b487-4542-9174-862708674b4c-maas-api-service-ca\") pod \"747b02b0-b487-4542-9174-862708674b4c\" (UID: \"747b02b0-b487-4542-9174-862708674b4c\") " Oct 30 19:30:19.098446 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.098423 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/747b02b0-b487-4542-9174-862708674b4c-maas-api-service-ca" pod "747b02b0-b487-4542-9174-862708674b4c" (UID: "747b02b0-b487-4542-9174-862708674b4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:19.100496 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.100463 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747b02b0-b487-4542-9174-862708674b4c-kube-api-access-vqcps" pod "747b02b0-b487-4542-9174-862708674b4c" (UID: "747b02b0-b487-4542-9174-862708674b4c"). InnerVolumeSpecName "kube-api-access-vqcps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:19.169606 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.169558 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7106-bf3302"] Oct 30 19:30:19.170039 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170024 4531 cpu_manager.go:414] "containerMap: removing container" podUID="5972ca80-ec09-4b6a-be76-847b415f88fb" containerName="cleanup" Oct 30 19:30:19.170102 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170040 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5972ca80-ec09-4b6a-be76-847b415f88fb" containerName="cleanup" Oct 30 19:30:19.170102 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170053 4531 cpu_manager.go:414] "containerMap: removing container" podUID="15436c31-541b-4441-b020-dc795e1387aa" containerName="cleanup" Oct 30 19:30:19.170102 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170058 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15436c31-541b-4441-b020-dc795e1387aa" containerName="cleanup" Oct 30 19:30:19.170102 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170071 4531 cpu_manager.go:414] "containerMap: removing container" podUID="5972ca80-ec09-4b6a-be76-847b415f88fb" containerName="cleanup" Oct 30 19:30:19.170102 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170075 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5972ca80-ec09-4b6a-be76-847b415f88fb" containerName="cleanup" Oct 30 19:30:19.170102 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170084 4531 cpu_manager.go:414] "containerMap: removing container" podUID="8d3ce71a-6c7f-4aca-829e-10b51893d417" containerName="cleanup" Oct 30 19:30:19.170102 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170088 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d3ce71a-6c7f-4aca-829e-10b51893d417" containerName="cleanup" Oct 30 19:30:19.170102 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170095 4531 cpu_manager.go:414] "containerMap: removing container" podUID="824fa435-1ea3-424e-9c73-5bbad49280b9" containerName="test-curl-7106-5c59ab" Oct 30 19:30:19.170102 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170099 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824fa435-1ea3-424e-9c73-5bbad49280b9" containerName="test-curl-7106-5c59ab" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170108 4531 cpu_manager.go:414] "containerMap: removing container" podUID="5972ca80-ec09-4b6a-be76-847b415f88fb" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170112 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5972ca80-ec09-4b6a-be76-847b415f88fb" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170118 4531 cpu_manager.go:414] "containerMap: removing container" podUID="0a549c34-f860-401d-a5e6-9c45f015cb54" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170123 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a549c34-f860-401d-a5e6-9c45f015cb54" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170144 4531 cpu_manager.go:414] "containerMap: removing container" podUID="15436c31-541b-4441-b020-dc795e1387aa" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170151 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15436c31-541b-4441-b020-dc795e1387aa" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170160 4531 cpu_manager.go:414] "containerMap: removing container" podUID="0a549c34-f860-401d-a5e6-9c45f015cb54" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170164 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a549c34-f860-401d-a5e6-9c45f015cb54" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170170 4531 cpu_manager.go:414] "containerMap: removing container" podUID="65f6ffce-2030-4be2-bf0a-cf4a56dec691" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170173 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f6ffce-2030-4be2-bf0a-cf4a56dec691" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170181 4531 cpu_manager.go:414] "containerMap: removing container" podUID="65f6ffce-2030-4be2-bf0a-cf4a56dec691" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170186 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f6ffce-2030-4be2-bf0a-cf4a56dec691" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170194 4531 cpu_manager.go:414] "containerMap: removing container" podUID="747b02b0-b487-4542-9174-862708674b4c" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170198 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="747b02b0-b487-4542-9174-862708674b4c" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170204 4531 cpu_manager.go:414] "containerMap: removing container" podUID="747b02b0-b487-4542-9174-862708674b4c" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170208 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="747b02b0-b487-4542-9174-862708674b4c" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170217 4531 cpu_manager.go:414] "containerMap: removing container" podUID="8d3ce71a-6c7f-4aca-829e-10b51893d417" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170221 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d3ce71a-6c7f-4aca-829e-10b51893d417" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170228 4531 cpu_manager.go:414] "containerMap: removing container" podUID="747b02b0-b487-4542-9174-862708674b4c" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170232 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="747b02b0-b487-4542-9174-862708674b4c" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170250 4531 cpu_manager.go:414] "containerMap: removing container" podUID="0a549c34-f860-401d-a5e6-9c45f015cb54" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170255 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a549c34-f860-401d-a5e6-9c45f015cb54" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170327 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d3ce71a-6c7f-4aca-829e-10b51893d417" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170335 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="824fa435-1ea3-424e-9c73-5bbad49280b9" containerName="test-curl-7106-5c59ab" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170342 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5972ca80-ec09-4b6a-be76-847b415f88fb" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170347 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a549c34-f860-401d-a5e6-9c45f015cb54" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170353 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="15436c31-541b-4441-b020-dc795e1387aa" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170358 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a549c34-f860-401d-a5e6-9c45f015cb54" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170363 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f6ffce-2030-4be2-bf0a-cf4a56dec691" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170368 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f6ffce-2030-4be2-bf0a-cf4a56dec691" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170374 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="747b02b0-b487-4542-9174-862708674b4c" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170379 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="747b02b0-b487-4542-9174-862708674b4c" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170385 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d3ce71a-6c7f-4aca-829e-10b51893d417" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170390 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="747b02b0-b487-4542-9174-862708674b4c" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170397 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f6ffce-2030-4be2-bf0a-cf4a56dec691" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170401 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d3ce71a-6c7f-4aca-829e-10b51893d417" containerName="cleanup" Oct 30 19:30:19.170392 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170407 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="15436c31-541b-4441-b020-dc795e1387aa" containerName="cleanup" Oct 30 19:30:19.171490 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.170413 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5972ca80-ec09-4b6a-be76-847b415f88fb" containerName="cleanup" Oct 30 19:30:19.173550 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.173528 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7106-bf3302" Oct 30 19:30:19.198759 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.198725 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65f6ffce-2030-4be2-bf0a-cf4a56dec691-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f6ffce-2030-4be2-bf0a-cf4a56dec691-maas-api-service-ca\") pod \"65f6ffce-2030-4be2-bf0a-cf4a56dec691\" (UID: \"65f6ffce-2030-4be2-bf0a-cf4a56dec691\") " Oct 30 19:30:19.198946 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.198787 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15436c31-541b-4441-b020-dc795e1387aa-kube-api-access-rp9kt\" (UniqueName: \"kubernetes.io/projected/15436c31-541b-4441-b020-dc795e1387aa-kube-api-access-rp9kt\") pod \"15436c31-541b-4441-b020-dc795e1387aa\" (UID: \"15436c31-541b-4441-b020-dc795e1387aa\") " Oct 30 19:30:19.198946 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.198831 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a549c34-f860-401d-a5e6-9c45f015cb54-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a549c34-f860-401d-a5e6-9c45f015cb54-maas-api-service-ca\") pod \"0a549c34-f860-401d-a5e6-9c45f015cb54\" (UID: \"0a549c34-f860-401d-a5e6-9c45f015cb54\") " Oct 30 19:30:19.198946 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.198880 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15436c31-541b-4441-b020-dc795e1387aa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15436c31-541b-4441-b020-dc795e1387aa-maas-api-service-ca\") pod \"15436c31-541b-4441-b020-dc795e1387aa\" (UID: \"15436c31-541b-4441-b020-dc795e1387aa\") " Oct 30 19:30:19.199187 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.199095 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5972ca80-ec09-4b6a-be76-847b415f88fb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5972ca80-ec09-4b6a-be76-847b415f88fb-maas-api-service-ca\") pod \"5972ca80-ec09-4b6a-be76-847b415f88fb\" (UID: \"5972ca80-ec09-4b6a-be76-847b415f88fb\") " Oct 30 19:30:19.199254 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.199211 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f6ffce-2030-4be2-bf0a-cf4a56dec691-maas-api-service-ca" pod "65f6ffce-2030-4be2-bf0a-cf4a56dec691" (UID: "65f6ffce-2030-4be2-bf0a-cf4a56dec691"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:19.199254 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.199242 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a549c34-f860-401d-a5e6-9c45f015cb54-kube-api-access-wsshr\" (UniqueName: \"kubernetes.io/projected/0a549c34-f860-401d-a5e6-9c45f015cb54-kube-api-access-wsshr\") pod \"0a549c34-f860-401d-a5e6-9c45f015cb54\" (UID: \"0a549c34-f860-401d-a5e6-9c45f015cb54\") " Oct 30 19:30:19.199358 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.199307 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15436c31-541b-4441-b020-dc795e1387aa-maas-api-service-ca" pod "15436c31-541b-4441-b020-dc795e1387aa" (UID: "15436c31-541b-4441-b020-dc795e1387aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:19.199358 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.199319 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a549c34-f860-401d-a5e6-9c45f015cb54-maas-api-service-ca" pod "0a549c34-f860-401d-a5e6-9c45f015cb54" (UID: "0a549c34-f860-401d-a5e6-9c45f015cb54"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:19.199484 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.199459 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5972ca80-ec09-4b6a-be76-847b415f88fb-maas-api-service-ca" pod "5972ca80-ec09-4b6a-be76-847b415f88fb" (UID: "5972ca80-ec09-4b6a-be76-847b415f88fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:19.199544 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.199505 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65f6ffce-2030-4be2-bf0a-cf4a56dec691-kube-api-access-x5gvt\" (UniqueName: \"kubernetes.io/projected/65f6ffce-2030-4be2-bf0a-cf4a56dec691-kube-api-access-x5gvt\") pod \"65f6ffce-2030-4be2-bf0a-cf4a56dec691\" (UID: \"65f6ffce-2030-4be2-bf0a-cf4a56dec691\") " Oct 30 19:30:19.199600 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.199560 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5972ca80-ec09-4b6a-be76-847b415f88fb-kube-api-access-8dz76\" (UniqueName: \"kubernetes.io/projected/5972ca80-ec09-4b6a-be76-847b415f88fb-kube-api-access-8dz76\") pod \"5972ca80-ec09-4b6a-be76-847b415f88fb\" (UID: \"5972ca80-ec09-4b6a-be76-847b415f88fb\") " Oct 30 19:30:19.200074 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.200045 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqcps\" (UniqueName: \"kubernetes.io/projected/747b02b0-b487-4542-9174-862708674b4c-kube-api-access-vqcps\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:19.200256 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.200185 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f6ffce-2030-4be2-bf0a-cf4a56dec691-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:19.200349 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.200261 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/747b02b0-b487-4542-9174-862708674b4c-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:19.200349 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.200275 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a549c34-f860-401d-a5e6-9c45f015cb54-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:19.200459 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.200319 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15436c31-541b-4441-b020-dc795e1387aa-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:19.200459 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.200407 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5972ca80-ec09-4b6a-be76-847b415f88fb-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:19.201664 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.201636 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15436c31-541b-4441-b020-dc795e1387aa-kube-api-access-rp9kt" pod "15436c31-541b-4441-b020-dc795e1387aa" (UID: "15436c31-541b-4441-b020-dc795e1387aa"). InnerVolumeSpecName "kube-api-access-rp9kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:19.201827 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.201763 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a549c34-f860-401d-a5e6-9c45f015cb54-kube-api-access-wsshr" pod "0a549c34-f860-401d-a5e6-9c45f015cb54" (UID: "0a549c34-f860-401d-a5e6-9c45f015cb54"). InnerVolumeSpecName "kube-api-access-wsshr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:19.202023 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.201997 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5972ca80-ec09-4b6a-be76-847b415f88fb-kube-api-access-8dz76" pod "5972ca80-ec09-4b6a-be76-847b415f88fb" (UID: "5972ca80-ec09-4b6a-be76-847b415f88fb"). InnerVolumeSpecName "kube-api-access-8dz76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:19.202191 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.202158 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f6ffce-2030-4be2-bf0a-cf4a56dec691-kube-api-access-x5gvt" pod "65f6ffce-2030-4be2-bf0a-cf4a56dec691" (UID: "65f6ffce-2030-4be2-bf0a-cf4a56dec691"). InnerVolumeSpecName "kube-api-access-x5gvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:19.229297 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.229253 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7106-bf3302"] Oct 30 19:30:19.301884 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.301841 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsxm2\" (UniqueName: \"kubernetes.io/projected/346fd946-d302-4fc0-af78-813258628d87-kube-api-access-bsxm2\") pod \"test-curl-7106-bf3302\" (UID: \"346fd946-d302-4fc0-af78-813258628d87\") " pod="opendatahub/test-curl-7106-bf3302" Oct 30 19:30:19.302106 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.301971 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5gvt\" (UniqueName: \"kubernetes.io/projected/65f6ffce-2030-4be2-bf0a-cf4a56dec691-kube-api-access-x5gvt\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:19.302106 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.301985 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dz76\" (UniqueName: \"kubernetes.io/projected/5972ca80-ec09-4b6a-be76-847b415f88fb-kube-api-access-8dz76\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:19.302106 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.301994 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp9kt\" (UniqueName: \"kubernetes.io/projected/15436c31-541b-4441-b020-dc795e1387aa-kube-api-access-rp9kt\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:19.302106 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.302004 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsshr\" (UniqueName: \"kubernetes.io/projected/0a549c34-f860-401d-a5e6-9c45f015cb54-kube-api-access-wsshr\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:19.403367 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.403317 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsxm2\" (UniqueName: \"kubernetes.io/projected/346fd946-d302-4fc0-af78-813258628d87-kube-api-access-bsxm2\") pod \"test-curl-7106-bf3302\" (UID: \"346fd946-d302-4fc0-af78-813258628d87\") " pod="opendatahub/test-curl-7106-bf3302" Oct 30 19:30:19.465230 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.465188 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsxm2\" (UniqueName: \"kubernetes.io/projected/346fd946-d302-4fc0-af78-813258628d87-kube-api-access-bsxm2\") pod \"test-curl-7106-bf3302\" (UID: \"346fd946-d302-4fc0-af78-813258628d87\") " pod="opendatahub/test-curl-7106-bf3302" Oct 30 19:30:19.501302 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.501260 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7106-bf3302" Oct 30 19:30:19.537891 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.537852 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb"] Oct 30 19:30:19.541963 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.541926 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a56c7c-29889810-sgzsb"] Oct 30 19:30:19.767298 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.767264 4531 generic.go:356] "Generic (PLEG): container finished" podID="10065963-ac42-42d0-902e-3ec7e87ebd06" containerID="ec56c817f0e2c0645808a89d2ee176a3feb097cfe4a9e22a7b54b709049f63b2" exitCode=60 Oct 30 19:30:19.767841 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.767415 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt" event={"ID":"10065963-ac42-42d0-902e-3ec7e87ebd06","Type":"ContainerDied","Data":"ec56c817f0e2c0645808a89d2ee176a3feb097cfe4a9e22a7b54b709049f63b2"} Oct 30 19:30:19.767841 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.767484 4531 scope.go:122] "RemoveContainer" containerID="c865e8cb697f1fd39b883c456e9b929f7e1fc214e40602497c2f25c185a21caa" Oct 30 19:30:19.770088 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.770060 4531 scope.go:122] "RemoveContainer" containerID="ec56c817f0e2c0645808a89d2ee176a3feb097cfe4a9e22a7b54b709049f63b2" Oct 30 19:30:19.770541 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:19.770505 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt_odh-ai-gateway-infra(10065963-ac42-42d0-902e-3ec7e87ebd06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt" podUID="10065963-ac42-42d0-902e-3ec7e87ebd06" Oct 30 19:30:19.771952 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.771919 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w" Oct 30 19:30:19.772082 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.771934 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w" event={"ID":"0a549c34-f860-401d-a5e6-9c45f015cb54","Type":"ContainerDied","Data":"75cb69c01ff29b22547af0a8fc9bdc96dfbb3316d2d29911adfb23447ba40eac"} Oct 30 19:30:19.774492 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.774447 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6" event={"ID":"541165d3-33cc-415c-9932-75e6502c70a8","Type":"ContainerStarted","Data":"6cac867f39dbe437029d3be777c4a567c13c5827ab60f33d281859764c49cd03"} Oct 30 19:30:19.777315 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.777282 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7" Oct 30 19:30:19.777527 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.777286 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7" event={"ID":"5972ca80-ec09-4b6a-be76-847b415f88fb","Type":"ContainerDied","Data":"0f3c7961e462fafc7891f288db6274a4a19c2b7be3be8a3176182ff9d911be52"} Oct 30 19:30:19.779220 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.779184 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l" event={"ID":"747b02b0-b487-4542-9174-862708674b4c","Type":"ContainerDied","Data":"3e087372270407edf343498ef145c5589863e08bb4b9b178d4087dae73bccb4e"} Oct 30 19:30:19.779376 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.779233 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l" Oct 30 19:30:19.781796 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.781727 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg" event={"ID":"65f6ffce-2030-4be2-bf0a-cf4a56dec691","Type":"ContainerDied","Data":"7b931927ef388fdb3f2ff08c02c48049cf9cef7bf6a7c23cb6f95367bad18c92"} Oct 30 19:30:19.781986 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.781857 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg" Oct 30 19:30:19.788082 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.788042 4531 scope.go:122] "RemoveContainer" containerID="023e434c8d6a2b67ef18118dbb4f5933767f3f5ff1cd9caea31066b6cf93df90" Oct 30 19:30:19.788618 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.788592 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9" Oct 30 19:30:19.788714 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.788626 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9" event={"ID":"15436c31-541b-4441-b020-dc795e1387aa","Type":"ContainerDied","Data":"30eb3363637d8dde29ef6fd1598c9d94edfe5f66fb29e2091dac7889b4a41c3b"} Oct 30 19:30:19.803485 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.803458 4531 scope.go:122] "RemoveContainer" containerID="b43ed4da879787967cb8efc667923330ba0d3c56f76cee5a79522a017dc270bf" Oct 30 19:30:19.815594 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.815100 4531 scope.go:122] "RemoveContainer" containerID="ccc208a585c45bfd5ab152e51f24c51164d7e62a845e161427534684a44ee224" Oct 30 19:30:19.827690 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.827665 4531 scope.go:122] "RemoveContainer" containerID="5d2b150e073958013817f81141baf8df6a226e89e1f30d2de92d76b0ec6771f0" Oct 30 19:30:19.839792 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.839753 4531 scope.go:122] "RemoveContainer" containerID="290bdf054c8d9babc109677a1b3e6b7f3d355ba95a5af5ae11fa5b5c7312d2bb" Oct 30 19:30:19.858827 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.858703 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt"] Oct 30 19:30:19.895289 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:30:19.894778 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod346fd946_d302_4fc0_af78_813258628d87.slice/crio-f5c127bee99b121c06ab7e904045dec70d621302d7db6080fb7d27f3181185b9 WatchSource:0}: Error finding container f5c127bee99b121c06ab7e904045dec70d621302d7db6080fb7d27f3181185b9: Status 404 returned error can't find the container with id f5c127bee99b121c06ab7e904045dec70d621302d7db6080fb7d27f3181185b9 Oct 30 19:30:19.895289 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.894955 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7106-bf3302"] Oct 30 19:30:19.914927 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:19.914885 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9" Oct 30 19:30:20.010373 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:20.010335 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f45f6f0-a99e-4381-a553-270bdb2f7621-kube-api-access-r4cnv\" (UniqueName: \"kubernetes.io/projected/6f45f6f0-a99e-4381-a553-270bdb2f7621-kube-api-access-r4cnv\") pod \"6f45f6f0-a99e-4381-a553-270bdb2f7621\" (UID: \"6f45f6f0-a99e-4381-a553-270bdb2f7621\") " Oct 30 19:30:20.010538 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:20.010413 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f45f6f0-a99e-4381-a553-270bdb2f7621-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f45f6f0-a99e-4381-a553-270bdb2f7621-maas-api-service-ca\") pod \"6f45f6f0-a99e-4381-a553-270bdb2f7621\" (UID: \"6f45f6f0-a99e-4381-a553-270bdb2f7621\") " Oct 30 19:30:20.010878 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:20.010845 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f45f6f0-a99e-4381-a553-270bdb2f7621-maas-api-service-ca" pod "6f45f6f0-a99e-4381-a553-270bdb2f7621" (UID: "6f45f6f0-a99e-4381-a553-270bdb2f7621"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:20.012993 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:20.012959 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f45f6f0-a99e-4381-a553-270bdb2f7621-kube-api-access-r4cnv" pod "6f45f6f0-a99e-4381-a553-270bdb2f7621" (UID: "6f45f6f0-a99e-4381-a553-270bdb2f7621"). InnerVolumeSpecName "kube-api-access-r4cnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:20.100228 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:20.100119 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d3ce71a-6c7f-4aca-829e-10b51893d417" path="/var/lib/kubelet/pods/8d3ce71a-6c7f-4aca-829e-10b51893d417/volumes" Oct 30 19:30:20.112082 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:20.112042 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4cnv\" (UniqueName: \"kubernetes.io/projected/6f45f6f0-a99e-4381-a553-270bdb2f7621-kube-api-access-r4cnv\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:20.112082 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:20.112078 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f45f6f0-a99e-4381-a553-270bdb2f7621-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:20.434830 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:20.434717 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-5c59ab\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="824fa435-1ea3-424e-9c73-5bbad49280b9" pod="opendatahub/test-curl-7106-5c59ab" Oct 30 19:30:20.665116 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:20.665072 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w"] Oct 30 19:30:20.675952 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:20.675919 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-19f6ea-29889810-2tg4w"] Oct 30 19:30:20.814582 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:20.814537 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7106-bf3302" event={"ID":"346fd946-d302-4fc0-af78-813258628d87","Type":"ContainerStarted","Data":"b78f7947c22407aad1ee290e184fa3a83697709c4d9cdc2c747c47353c94534b"} Oct 30 19:30:20.815034 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:20.814600 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7106-bf3302" event={"ID":"346fd946-d302-4fc0-af78-813258628d87","Type":"ContainerStarted","Data":"f5c127bee99b121c06ab7e904045dec70d621302d7db6080fb7d27f3181185b9"} Oct 30 19:30:20.817036 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:20.816999 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9" Oct 30 19:30:20.817241 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:20.817022 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9" event={"ID":"6f45f6f0-a99e-4381-a553-270bdb2f7621","Type":"ContainerDied","Data":"dc7fa253b5e0125fdadd5a72892bde7533d867006bff71a3bcf93c0d608c95f4"} Oct 30 19:30:20.817241 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:20.817156 4531 scope.go:122] "RemoveContainer" containerID="0581696f55496a8da6e323eb1d22aa0d32b1ad58eb676dd39ec191289f671991" Oct 30 19:30:20.826812 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:20.826769 4531 generic.go:356] "Generic (PLEG): container finished" podID="541165d3-33cc-415c-9932-75e6502c70a8" containerID="6cac867f39dbe437029d3be777c4a567c13c5827ab60f33d281859764c49cd03" exitCode=60 Oct 30 19:30:20.827001 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:20.826824 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6" event={"ID":"541165d3-33cc-415c-9932-75e6502c70a8","Type":"ContainerDied","Data":"6cac867f39dbe437029d3be777c4a567c13c5827ab60f33d281859764c49cd03"} Oct 30 19:30:20.827326 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:20.827301 4531 scope.go:122] "RemoveContainer" containerID="6cac867f39dbe437029d3be777c4a567c13c5827ab60f33d281859764c49cd03" Oct 30 19:30:20.827913 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:20.827882 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6_odh-ai-gateway-infra(541165d3-33cc-415c-9932-75e6502c70a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6" podUID="541165d3-33cc-415c-9932-75e6502c70a8" Oct 30 19:30:20.833364 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:20.833337 4531 scope.go:122] "RemoveContainer" containerID="d1347626378fc58158f46e3726a6ac554efd2110cefe5ee0d44afbd87b232b80" Oct 30 19:30:20.901439 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:20.901403 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg"] Oct 30 19:30:20.912763 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:20.912723 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-dc5cdd-29889810-wp8mg"] Oct 30 19:30:20.996107 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:20.996075 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt" Oct 30 19:30:21.024644 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.024593 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7"] Oct 30 19:30:21.028036 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.028013 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4a7a52-29889810-2lxx7"] Oct 30 19:30:21.126050 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.125962 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10065963-ac42-42d0-902e-3ec7e87ebd06-kube-api-access-7hv7m\" (UniqueName: \"kubernetes.io/projected/10065963-ac42-42d0-902e-3ec7e87ebd06-kube-api-access-7hv7m\") pod \"10065963-ac42-42d0-902e-3ec7e87ebd06\" (UID: \"10065963-ac42-42d0-902e-3ec7e87ebd06\") " Oct 30 19:30:21.126271 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.126057 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10065963-ac42-42d0-902e-3ec7e87ebd06-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10065963-ac42-42d0-902e-3ec7e87ebd06-maas-api-service-ca\") pod \"10065963-ac42-42d0-902e-3ec7e87ebd06\" (UID: \"10065963-ac42-42d0-902e-3ec7e87ebd06\") " Oct 30 19:30:21.126446 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.126422 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10065963-ac42-42d0-902e-3ec7e87ebd06-maas-api-service-ca" pod "10065963-ac42-42d0-902e-3ec7e87ebd06" (UID: "10065963-ac42-42d0-902e-3ec7e87ebd06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:21.128266 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.128240 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10065963-ac42-42d0-902e-3ec7e87ebd06-kube-api-access-7hv7m" pod "10065963-ac42-42d0-902e-3ec7e87ebd06" (UID: "10065963-ac42-42d0-902e-3ec7e87ebd06"). InnerVolumeSpecName "kube-api-access-7hv7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:21.140663 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.140642 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9"] Oct 30 19:30:21.149745 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.149710 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f5dfb8-29889810-zdpd9"] Oct 30 19:30:21.220438 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.220407 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7106-5c59ab" Oct 30 19:30:21.227497 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.227472 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hv7m\" (UniqueName: \"kubernetes.io/projected/10065963-ac42-42d0-902e-3ec7e87ebd06-kube-api-access-7hv7m\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:21.227497 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.227497 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10065963-ac42-42d0-902e-3ec7e87ebd06-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:21.266573 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.266540 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l"] Oct 30 19:30:21.274920 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.274887 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889810-ggz9l"] Oct 30 19:30:21.328072 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.328033 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824fa435-1ea3-424e-9c73-5bbad49280b9-kube-api-access-sgl2l\" (UniqueName: \"kubernetes.io/projected/824fa435-1ea3-424e-9c73-5bbad49280b9-kube-api-access-sgl2l\") pod \"824fa435-1ea3-424e-9c73-5bbad49280b9\" (UID: \"824fa435-1ea3-424e-9c73-5bbad49280b9\") " Oct 30 19:30:21.331407 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.331360 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824fa435-1ea3-424e-9c73-5bbad49280b9-kube-api-access-sgl2l" pod "824fa435-1ea3-424e-9c73-5bbad49280b9" (UID: "824fa435-1ea3-424e-9c73-5bbad49280b9"). InnerVolumeSpecName "kube-api-access-sgl2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:21.429974 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.429886 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgl2l\" (UniqueName: \"kubernetes.io/projected/824fa435-1ea3-424e-9c73-5bbad49280b9-kube-api-access-sgl2l\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:21.620768 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.620727 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9"] Oct 30 19:30:21.628494 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.628463 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d4f84d-29889810-sdxz9"] Oct 30 19:30:21.836609 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.836570 4531 generic.go:356] "Generic (PLEG): container finished" podID="824fa435-1ea3-424e-9c73-5bbad49280b9" containerID="901ca5943ade131beef9ef604cedb9826da1b25888b5fddcd79531437c4ac36d" exitCode=137 Oct 30 19:30:21.837153 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.836627 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7106-5c59ab" Oct 30 19:30:21.837153 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.836659 4531 scope.go:122] "RemoveContainer" containerID="901ca5943ade131beef9ef604cedb9826da1b25888b5fddcd79531437c4ac36d" Oct 30 19:30:21.839398 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.839369 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt" event={"ID":"10065963-ac42-42d0-902e-3ec7e87ebd06","Type":"ContainerDied","Data":"4bbb8506aed02daf18001c76d1f736ea95cb43b2b89e3f29712890e2de038637"} Oct 30 19:30:21.839543 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.839400 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt" Oct 30 19:30:21.848800 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.848758 4531 scope.go:122] "RemoveContainer" containerID="901ca5943ade131beef9ef604cedb9826da1b25888b5fddcd79531437c4ac36d" Oct 30 19:30:21.849228 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:21.849198 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"901ca5943ade131beef9ef604cedb9826da1b25888b5fddcd79531437c4ac36d\": container with ID starting with 901ca5943ade131beef9ef604cedb9826da1b25888b5fddcd79531437c4ac36d not found: ID does not exist" containerID="901ca5943ade131beef9ef604cedb9826da1b25888b5fddcd79531437c4ac36d" Oct 30 19:30:21.849359 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.849239 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"901ca5943ade131beef9ef604cedb9826da1b25888b5fddcd79531437c4ac36d"} err="failed to get container status \"901ca5943ade131beef9ef604cedb9826da1b25888b5fddcd79531437c4ac36d\": rpc error: code = NotFound desc = could not find container \"901ca5943ade131beef9ef604cedb9826da1b25888b5fddcd79531437c4ac36d\": container with ID starting with 901ca5943ade131beef9ef604cedb9826da1b25888b5fddcd79531437c4ac36d not found: ID does not exist" Oct 30 19:30:21.849359 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.849266 4531 scope.go:122] "RemoveContainer" containerID="ec56c817f0e2c0645808a89d2ee176a3feb097cfe4a9e22a7b54b709049f63b2" Oct 30 19:30:21.860489 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.860428 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7106-bf3302" podStartSLOduration=2.860411462 podStartE2EDuration="2.860411462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:21.859528761 +0000 UTC m=+3381.886633721" watchObservedRunningTime="2026-10-30 19:30:21.860411462 +0000 UTC m=+3381.887516422" Oct 30 19:30:21.944206 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.944168 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x"] Oct 30 19:30:21.972115 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.972045 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt"] Oct 30 19:30:21.976913 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:21.976869 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-23302c-29889810-pzpmt"] Oct 30 19:30:22.025679 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:22.025633 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-5c59ab\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="824fa435-1ea3-424e-9c73-5bbad49280b9" pod="opendatahub/test-curl-7106-5c59ab" Oct 30 19:30:22.077201 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:22.077148 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-5c59ab\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="824fa435-1ea3-424e-9c73-5bbad49280b9" pod="opendatahub/test-curl-7106-5c59ab" Oct 30 19:30:22.100327 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:22.100251 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a549c34-f860-401d-a5e6-9c45f015cb54" path="/var/lib/kubelet/pods/0a549c34-f860-401d-a5e6-9c45f015cb54/volumes" Oct 30 19:30:22.100693 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:22.100679 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10065963-ac42-42d0-902e-3ec7e87ebd06" path="/var/lib/kubelet/pods/10065963-ac42-42d0-902e-3ec7e87ebd06/volumes" Oct 30 19:30:22.101043 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:22.101031 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15436c31-541b-4441-b020-dc795e1387aa" path="/var/lib/kubelet/pods/15436c31-541b-4441-b020-dc795e1387aa/volumes" Oct 30 19:30:22.101793 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:22.101774 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5972ca80-ec09-4b6a-be76-847b415f88fb" path="/var/lib/kubelet/pods/5972ca80-ec09-4b6a-be76-847b415f88fb/volumes" Oct 30 19:30:22.102152 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:22.102121 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65f6ffce-2030-4be2-bf0a-cf4a56dec691" path="/var/lib/kubelet/pods/65f6ffce-2030-4be2-bf0a-cf4a56dec691/volumes" Oct 30 19:30:22.102499 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:22.102486 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f45f6f0-a99e-4381-a553-270bdb2f7621" path="/var/lib/kubelet/pods/6f45f6f0-a99e-4381-a553-270bdb2f7621/volumes" Oct 30 19:30:22.103135 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:22.103112 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="747b02b0-b487-4542-9174-862708674b4c" path="/var/lib/kubelet/pods/747b02b0-b487-4542-9174-862708674b4c/volumes" Oct 30 19:30:22.103461 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:22.103447 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="824fa435-1ea3-424e-9c73-5bbad49280b9" path="/var/lib/kubelet/pods/824fa435-1ea3-424e-9c73-5bbad49280b9/volumes" Oct 30 19:30:22.413494 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:22.412959 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6"] Oct 30 19:30:22.572883 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:22.572849 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6" Oct 30 19:30:22.645870 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:22.645833 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4"] Oct 30 19:30:22.656495 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:22.656449 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f5dfb8-54gm4"] Oct 30 19:30:22.744788 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:22.744747 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/541165d3-33cc-415c-9932-75e6502c70a8-kube-api-access-4gfw9\" (UniqueName: \"kubernetes.io/projected/541165d3-33cc-415c-9932-75e6502c70a8-kube-api-access-4gfw9\") pod \"541165d3-33cc-415c-9932-75e6502c70a8\" (UID: \"541165d3-33cc-415c-9932-75e6502c70a8\") " Oct 30 19:30:22.744971 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:22.744940 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/541165d3-33cc-415c-9932-75e6502c70a8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/541165d3-33cc-415c-9932-75e6502c70a8-maas-api-service-ca\") pod \"541165d3-33cc-415c-9932-75e6502c70a8\" (UID: \"541165d3-33cc-415c-9932-75e6502c70a8\") " Oct 30 19:30:22.745335 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:22.745310 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/541165d3-33cc-415c-9932-75e6502c70a8-maas-api-service-ca" pod "541165d3-33cc-415c-9932-75e6502c70a8" (UID: "541165d3-33cc-415c-9932-75e6502c70a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:22.747447 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:22.747414 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/541165d3-33cc-415c-9932-75e6502c70a8-kube-api-access-4gfw9" pod "541165d3-33cc-415c-9932-75e6502c70a8" (UID: "541165d3-33cc-415c-9932-75e6502c70a8"). InnerVolumeSpecName "kube-api-access-4gfw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:22.846065 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:22.846034 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gfw9\" (UniqueName: \"kubernetes.io/projected/541165d3-33cc-415c-9932-75e6502c70a8-kube-api-access-4gfw9\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:22.846065 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:22.846061 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/541165d3-33cc-415c-9932-75e6502c70a8-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:22.847659 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:22.847630 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6" Oct 30 19:30:22.847659 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:22.847636 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6" event={"ID":"541165d3-33cc-415c-9932-75e6502c70a8","Type":"ContainerDied","Data":"f73b750bfc6ea29b1b1f80bcd51e80eef94ef804c06bbcbab483d2059b867483"} Oct 30 19:30:22.847866 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:22.847751 4531 scope.go:122] "RemoveContainer" containerID="6cac867f39dbe437029d3be777c4a567c13c5827ab60f33d281859764c49cd03" Oct 30 19:30:23.013208 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.013103 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6"] Oct 30 19:30:23.020413 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.020380 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3b6013-29889810-6hbp6"] Oct 30 19:30:23.486621 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.486511 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7106-bf3302"] Oct 30 19:30:23.486799 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.486719 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7106-bf3302" podUID="346fd946-d302-4fc0-af78-813258628d87" containerName="test-curl-7106-bf3302" containerID="cri-o://b78f7947c22407aad1ee290e184fa3a83697709c4d9cdc2c747c47353c94534b" gracePeriod=2 Oct 30 19:30:23.491674 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.491637 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7106-bf3302"] Oct 30 19:30:23.537813 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:23.537758 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-bf3302\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="346fd946-d302-4fc0-af78-813258628d87" pod="opendatahub/test-curl-7106-bf3302" Oct 30 19:30:23.894123 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.894088 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7106-c3b1dd"] Oct 30 19:30:23.894724 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.894703 4531 cpu_manager.go:414] "containerMap: removing container" podUID="541165d3-33cc-415c-9932-75e6502c70a8" containerName="cleanup" Oct 30 19:30:23.894794 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.894728 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="541165d3-33cc-415c-9932-75e6502c70a8" containerName="cleanup" Oct 30 19:30:23.894794 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.894745 4531 cpu_manager.go:414] "containerMap: removing container" podUID="6f45f6f0-a99e-4381-a553-270bdb2f7621" containerName="cleanup" Oct 30 19:30:23.894794 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.894752 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f45f6f0-a99e-4381-a553-270bdb2f7621" containerName="cleanup" Oct 30 19:30:23.894794 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.894764 4531 cpu_manager.go:414] "containerMap: removing container" podUID="15436c31-541b-4441-b020-dc795e1387aa" containerName="cleanup" Oct 30 19:30:23.894794 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.894772 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15436c31-541b-4441-b020-dc795e1387aa" containerName="cleanup" Oct 30 19:30:23.894794 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.894789 4531 cpu_manager.go:414] "containerMap: removing container" podUID="65f6ffce-2030-4be2-bf0a-cf4a56dec691" containerName="cleanup" Oct 30 19:30:23.894794 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.894795 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f6ffce-2030-4be2-bf0a-cf4a56dec691" containerName="cleanup" Oct 30 19:30:23.895048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.894811 4531 cpu_manager.go:414] "containerMap: removing container" podUID="346fd946-d302-4fc0-af78-813258628d87" containerName="test-curl-7106-bf3302" Oct 30 19:30:23.895048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.894817 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="346fd946-d302-4fc0-af78-813258628d87" containerName="test-curl-7106-bf3302" Oct 30 19:30:23.895048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.894830 4531 cpu_manager.go:414] "containerMap: removing container" podUID="10065963-ac42-42d0-902e-3ec7e87ebd06" containerName="cleanup" Oct 30 19:30:23.895048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.894836 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10065963-ac42-42d0-902e-3ec7e87ebd06" containerName="cleanup" Oct 30 19:30:23.895048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.894847 4531 cpu_manager.go:414] "containerMap: removing container" podUID="541165d3-33cc-415c-9932-75e6502c70a8" containerName="cleanup" Oct 30 19:30:23.895048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.894853 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="541165d3-33cc-415c-9932-75e6502c70a8" containerName="cleanup" Oct 30 19:30:23.895048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.894863 4531 cpu_manager.go:414] "containerMap: removing container" podUID="8d3ce71a-6c7f-4aca-829e-10b51893d417" containerName="cleanup" Oct 30 19:30:23.895048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.894869 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d3ce71a-6c7f-4aca-829e-10b51893d417" containerName="cleanup" Oct 30 19:30:23.895048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.894886 4531 cpu_manager.go:414] "containerMap: removing container" podUID="541165d3-33cc-415c-9932-75e6502c70a8" containerName="cleanup" Oct 30 19:30:23.895048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.894891 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="541165d3-33cc-415c-9932-75e6502c70a8" containerName="cleanup" Oct 30 19:30:23.895048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.894908 4531 cpu_manager.go:414] "containerMap: removing container" podUID="6f45f6f0-a99e-4381-a553-270bdb2f7621" containerName="cleanup" Oct 30 19:30:23.895048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.894914 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f45f6f0-a99e-4381-a553-270bdb2f7621" containerName="cleanup" Oct 30 19:30:23.895048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.894929 4531 cpu_manager.go:414] "containerMap: removing container" podUID="10065963-ac42-42d0-902e-3ec7e87ebd06" containerName="cleanup" Oct 30 19:30:23.895048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.894935 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10065963-ac42-42d0-902e-3ec7e87ebd06" containerName="cleanup" Oct 30 19:30:23.895048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.894955 4531 cpu_manager.go:414] "containerMap: removing container" podUID="6f45f6f0-a99e-4381-a553-270bdb2f7621" containerName="cleanup" Oct 30 19:30:23.895048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.894960 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f45f6f0-a99e-4381-a553-270bdb2f7621" containerName="cleanup" Oct 30 19:30:23.895048 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.895053 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="541165d3-33cc-415c-9932-75e6502c70a8" containerName="cleanup" Oct 30 19:30:23.895639 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.895066 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="541165d3-33cc-415c-9932-75e6502c70a8" containerName="cleanup" Oct 30 19:30:23.895639 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.895077 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="10065963-ac42-42d0-902e-3ec7e87ebd06" containerName="cleanup" Oct 30 19:30:23.895639 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.895086 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f45f6f0-a99e-4381-a553-270bdb2f7621" containerName="cleanup" Oct 30 19:30:23.895639 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.895094 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="10065963-ac42-42d0-902e-3ec7e87ebd06" containerName="cleanup" Oct 30 19:30:23.895639 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.895102 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="15436c31-541b-4441-b020-dc795e1387aa" containerName="cleanup" Oct 30 19:30:23.895639 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.895110 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f45f6f0-a99e-4381-a553-270bdb2f7621" containerName="cleanup" Oct 30 19:30:23.895639 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.895119 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5972ca80-ec09-4b6a-be76-847b415f88fb" containerName="cleanup" Oct 30 19:30:23.895639 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.895147 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a549c34-f860-401d-a5e6-9c45f015cb54" containerName="cleanup" Oct 30 19:30:23.895639 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.895157 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="346fd946-d302-4fc0-af78-813258628d87" containerName="test-curl-7106-bf3302" Oct 30 19:30:23.899895 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.899863 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7106-c3b1dd" Oct 30 19:30:23.953655 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:23.953621 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7106-c3b1dd"] Oct 30 19:30:24.057149 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:24.057092 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twchx\" (UniqueName: \"kubernetes.io/projected/022277ec-105e-4bec-97d5-d2de22ec5ddc-kube-api-access-twchx\") pod \"test-curl-7106-c3b1dd\" (UID: \"022277ec-105e-4bec-97d5-d2de22ec5ddc\") " pod="opendatahub/test-curl-7106-c3b1dd" Oct 30 19:30:24.064750 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:24.064695 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-bf3302\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="346fd946-d302-4fc0-af78-813258628d87" pod="opendatahub/test-curl-7106-bf3302" Oct 30 19:30:24.101768 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:24.101720 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42119ccd-547b-41b3-95ac-9443ad52dd6f" path="/var/lib/kubelet/pods/42119ccd-547b-41b3-95ac-9443ad52dd6f/volumes" Oct 30 19:30:24.102354 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:24.102332 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="541165d3-33cc-415c-9932-75e6502c70a8" path="/var/lib/kubelet/pods/541165d3-33cc-415c-9932-75e6502c70a8/volumes" Oct 30 19:30:24.158290 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:24.158192 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twchx\" (UniqueName: \"kubernetes.io/projected/022277ec-105e-4bec-97d5-d2de22ec5ddc-kube-api-access-twchx\") pod \"test-curl-7106-c3b1dd\" (UID: \"022277ec-105e-4bec-97d5-d2de22ec5ddc\") " pod="opendatahub/test-curl-7106-c3b1dd" Oct 30 19:30:24.220097 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:24.220061 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twchx\" (UniqueName: \"kubernetes.io/projected/022277ec-105e-4bec-97d5-d2de22ec5ddc-kube-api-access-twchx\") pod \"test-curl-7106-c3b1dd\" (UID: \"022277ec-105e-4bec-97d5-d2de22ec5ddc\") " pod="opendatahub/test-curl-7106-c3b1dd" Oct 30 19:30:24.515940 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:24.515898 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7106-c3b1dd" Oct 30 19:30:24.897422 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:24.897396 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7106-c3b1dd"] Oct 30 19:30:24.899517 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:30:24.899478 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod022277ec_105e_4bec_97d5_d2de22ec5ddc.slice/crio-85e5695cd91b1589ff846f10b51dd5c4c5af87f55f10bd14892a3fda283399ed WatchSource:0}: Error finding container 85e5695cd91b1589ff846f10b51dd5c4c5af87f55f10bd14892a3fda283399ed: Status 404 returned error can't find the container with id 85e5695cd91b1589ff846f10b51dd5c4c5af87f55f10bd14892a3fda283399ed Oct 30 19:30:25.774535 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:25.774501 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7106-bf3302" Oct 30 19:30:25.872104 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:25.872061 4531 generic.go:356] "Generic (PLEG): container finished" podID="346fd946-d302-4fc0-af78-813258628d87" containerID="b78f7947c22407aad1ee290e184fa3a83697709c4d9cdc2c747c47353c94534b" exitCode=137 Oct 30 19:30:25.872320 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:25.872110 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7106-bf3302" Oct 30 19:30:25.872320 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:25.872176 4531 scope.go:122] "RemoveContainer" containerID="b78f7947c22407aad1ee290e184fa3a83697709c4d9cdc2c747c47353c94534b" Oct 30 19:30:25.874027 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:25.874001 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7106-c3b1dd" event={"ID":"022277ec-105e-4bec-97d5-d2de22ec5ddc","Type":"ContainerStarted","Data":"0c932f2ddfbc1ffcc0a805142b30e3ebe0b7a7caf228d28759b106b2c8aa5ddc"} Oct 30 19:30:25.874157 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:25.874038 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7106-c3b1dd" event={"ID":"022277ec-105e-4bec-97d5-d2de22ec5ddc","Type":"ContainerStarted","Data":"85e5695cd91b1589ff846f10b51dd5c4c5af87f55f10bd14892a3fda283399ed"} Oct 30 19:30:25.877683 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:25.877660 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/346fd946-d302-4fc0-af78-813258628d87-kube-api-access-bsxm2\" (UniqueName: \"kubernetes.io/projected/346fd946-d302-4fc0-af78-813258628d87-kube-api-access-bsxm2\") pod \"346fd946-d302-4fc0-af78-813258628d87\" (UID: \"346fd946-d302-4fc0-af78-813258628d87\") " Oct 30 19:30:25.879872 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:25.879841 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/346fd946-d302-4fc0-af78-813258628d87-kube-api-access-bsxm2" pod "346fd946-d302-4fc0-af78-813258628d87" (UID: "346fd946-d302-4fc0-af78-813258628d87"). InnerVolumeSpecName "kube-api-access-bsxm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:25.882008 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:25.881980 4531 scope.go:122] "RemoveContainer" containerID="b78f7947c22407aad1ee290e184fa3a83697709c4d9cdc2c747c47353c94534b" Oct 30 19:30:25.882417 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:25.882393 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b78f7947c22407aad1ee290e184fa3a83697709c4d9cdc2c747c47353c94534b\": container with ID starting with b78f7947c22407aad1ee290e184fa3a83697709c4d9cdc2c747c47353c94534b not found: ID does not exist" containerID="b78f7947c22407aad1ee290e184fa3a83697709c4d9cdc2c747c47353c94534b" Oct 30 19:30:25.882489 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:25.882426 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b78f7947c22407aad1ee290e184fa3a83697709c4d9cdc2c747c47353c94534b"} err="failed to get container status \"b78f7947c22407aad1ee290e184fa3a83697709c4d9cdc2c747c47353c94534b\": rpc error: code = NotFound desc = could not find container \"b78f7947c22407aad1ee290e184fa3a83697709c4d9cdc2c747c47353c94534b\": container with ID starting with b78f7947c22407aad1ee290e184fa3a83697709c4d9cdc2c747c47353c94534b not found: ID does not exist" Oct 30 19:30:25.979681 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:25.979635 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsxm2\" (UniqueName: \"kubernetes.io/projected/346fd946-d302-4fc0-af78-813258628d87-kube-api-access-bsxm2\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:25.995015 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:25.994950 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7106-c3b1dd" podStartSLOduration=2.9949351379999998 podStartE2EDuration="2.994935138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:25.992060404 +0000 UTC m=+3386.019165376" watchObservedRunningTime="2026-10-30 19:30:25.994935138 +0000 UTC m=+3386.022040097" Oct 30 19:30:26.102830 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:26.102746 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="346fd946-d302-4fc0-af78-813258628d87" path="/var/lib/kubelet/pods/346fd946-d302-4fc0-af78-813258628d87/volumes" Oct 30 19:30:26.400744 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:26.400655 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:30:27.464354 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:27.464314 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7106-c3b1dd"] Oct 30 19:30:27.470408 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:27.470376 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7106-c3b1dd"] Oct 30 19:30:27.632267 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:27.632216 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7106-9d0e92"] Oct 30 19:30:27.632745 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:27.632723 4531 cpu_manager.go:414] "containerMap: removing container" podUID="022277ec-105e-4bec-97d5-d2de22ec5ddc" containerName="test-curl-7106-c3b1dd" Oct 30 19:30:27.632909 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:27.632748 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="022277ec-105e-4bec-97d5-d2de22ec5ddc" containerName="test-curl-7106-c3b1dd" Oct 30 19:30:27.632909 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:27.632784 4531 cpu_manager.go:414] "containerMap: removing container" podUID="10065963-ac42-42d0-902e-3ec7e87ebd06" containerName="cleanup" Oct 30 19:30:27.632909 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:27.632792 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10065963-ac42-42d0-902e-3ec7e87ebd06" containerName="cleanup" Oct 30 19:30:27.633071 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:27.632933 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f45f6f0-a99e-4381-a553-270bdb2f7621" containerName="cleanup" Oct 30 19:30:27.633071 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:27.632957 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="022277ec-105e-4bec-97d5-d2de22ec5ddc" containerName="test-curl-7106-c3b1dd" Oct 30 19:30:27.633071 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:27.632971 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="541165d3-33cc-415c-9932-75e6502c70a8" containerName="cleanup" Oct 30 19:30:27.633071 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:27.632981 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="10065963-ac42-42d0-902e-3ec7e87ebd06" containerName="cleanup" Oct 30 19:30:27.638073 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:27.638041 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7106-9d0e92" Oct 30 19:30:27.693348 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:27.693312 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7106-9d0e92"] Oct 30 19:30:27.700691 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:27.700638 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlzzp\" (UniqueName: \"kubernetes.io/projected/a6acdaee-969b-4087-9563-4ef8f8274df8-kube-api-access-nlzzp\") pod \"test-curl-7106-9d0e92\" (UID: \"a6acdaee-969b-4087-9563-4ef8f8274df8\") " pod="opendatahub/test-curl-7106-9d0e92" Oct 30 19:30:27.801401 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:27.801361 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlzzp\" (UniqueName: \"kubernetes.io/projected/a6acdaee-969b-4087-9563-4ef8f8274df8-kube-api-access-nlzzp\") pod \"test-curl-7106-9d0e92\" (UID: \"a6acdaee-969b-4087-9563-4ef8f8274df8\") " pod="opendatahub/test-curl-7106-9d0e92" Oct 30 19:30:27.862690 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:27.862657 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlzzp\" (UniqueName: \"kubernetes.io/projected/a6acdaee-969b-4087-9563-4ef8f8274df8-kube-api-access-nlzzp\") pod \"test-curl-7106-9d0e92\" (UID: \"a6acdaee-969b-4087-9563-4ef8f8274df8\") " pod="opendatahub/test-curl-7106-9d0e92" Oct 30 19:30:27.887487 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:27.887427 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7106-c3b1dd" podUID="022277ec-105e-4bec-97d5-d2de22ec5ddc" containerName="test-curl-7106-c3b1dd" containerID="cri-o://0c932f2ddfbc1ffcc0a805142b30e3ebe0b7a7caf228d28759b106b2c8aa5ddc" gracePeriod=2 Oct 30 19:30:27.938434 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:27.938388 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-c3b1dd\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="022277ec-105e-4bec-97d5-d2de22ec5ddc" pod="opendatahub/test-curl-7106-c3b1dd" Oct 30 19:30:27.950884 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:27.950851 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7106-9d0e92" Oct 30 19:30:28.330182 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:28.330157 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7106-9d0e92"] Oct 30 19:30:28.332414 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:30:28.332381 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6acdaee_969b_4087_9563_4ef8f8274df8.slice/crio-b0d9363b96ce7cd6fb371274d066967d0870d0bc286498737c609fbf7bc88767 WatchSource:0}: Error finding container b0d9363b96ce7cd6fb371274d066967d0870d0bc286498737c609fbf7bc88767: Status 404 returned error can't find the container with id b0d9363b96ce7cd6fb371274d066967d0870d0bc286498737c609fbf7bc88767 Oct 30 19:30:28.895473 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:28.895425 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7106-9d0e92" event={"ID":"a6acdaee-969b-4087-9563-4ef8f8274df8","Type":"ContainerStarted","Data":"7fb7f4efcb8729a71d30c9ee579e6c8343ab65eed82c5b7c2c322c5c239fb0f4"} Oct 30 19:30:28.895924 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:28.895484 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7106-9d0e92" event={"ID":"a6acdaee-969b-4087-9563-4ef8f8274df8","Type":"ContainerStarted","Data":"b0d9363b96ce7cd6fb371274d066967d0870d0bc286498737c609fbf7bc88767"} Oct 30 19:30:29.012793 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:29.012714 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7106-9d0e92" podStartSLOduration=2.012693266 podStartE2EDuration="2.012693266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:29.010733732 +0000 UTC m=+3389.037838695" watchObservedRunningTime="2026-10-30 19:30:29.012693266 +0000 UTC m=+3389.039798227" Oct 30 19:30:30.130158 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:30.124242 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7106-c3b1dd" Oct 30 19:30:30.175457 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:30.175363 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-c3b1dd\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="022277ec-105e-4bec-97d5-d2de22ec5ddc" pod="opendatahub/test-curl-7106-c3b1dd" Oct 30 19:30:30.229997 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:30.229956 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/022277ec-105e-4bec-97d5-d2de22ec5ddc-kube-api-access-twchx\" (UniqueName: \"kubernetes.io/projected/022277ec-105e-4bec-97d5-d2de22ec5ddc-kube-api-access-twchx\") pod \"022277ec-105e-4bec-97d5-d2de22ec5ddc\" (UID: \"022277ec-105e-4bec-97d5-d2de22ec5ddc\") " Oct 30 19:30:30.240794 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:30.240742 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/022277ec-105e-4bec-97d5-d2de22ec5ddc-kube-api-access-twchx" pod "022277ec-105e-4bec-97d5-d2de22ec5ddc" (UID: "022277ec-105e-4bec-97d5-d2de22ec5ddc"). InnerVolumeSpecName "kube-api-access-twchx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:30.331081 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:30.331038 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twchx\" (UniqueName: \"kubernetes.io/projected/022277ec-105e-4bec-97d5-d2de22ec5ddc-kube-api-access-twchx\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:30.558718 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:30.558663 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7106-9d0e92"] Oct 30 19:30:30.563682 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:30.563643 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7106-9d0e92"] Oct 30 19:30:30.725707 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:30.725666 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7106-749cf7"] Oct 30 19:30:30.726108 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:30.726093 4531 cpu_manager.go:414] "containerMap: removing container" podUID="a6acdaee-969b-4087-9563-4ef8f8274df8" containerName="test-curl-7106-9d0e92" Oct 30 19:30:30.726194 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:30.726109 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6acdaee-969b-4087-9563-4ef8f8274df8" containerName="test-curl-7106-9d0e92" Oct 30 19:30:30.726263 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:30.726252 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6acdaee-969b-4087-9563-4ef8f8274df8" containerName="test-curl-7106-9d0e92" Oct 30 19:30:30.730821 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:30.730774 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7106-749cf7" Oct 30 19:30:30.782474 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:30.782419 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-c3b1dd\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="022277ec-105e-4bec-97d5-d2de22ec5ddc" pod="opendatahub/test-curl-7106-c3b1dd" Oct 30 19:30:30.785922 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:30.785892 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7106-749cf7"] Oct 30 19:30:30.837865 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:30.837778 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbfmz\" (UniqueName: \"kubernetes.io/projected/316519ba-9ca9-42f9-9bdd-659fe52ee33c-kube-api-access-vbfmz\") pod \"test-curl-7106-749cf7\" (UID: \"316519ba-9ca9-42f9-9bdd-659fe52ee33c\") " pod="opendatahub/test-curl-7106-749cf7" Oct 30 19:30:30.910095 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:30.910063 4531 generic.go:356] "Generic (PLEG): container finished" podID="022277ec-105e-4bec-97d5-d2de22ec5ddc" containerID="0c932f2ddfbc1ffcc0a805142b30e3ebe0b7a7caf228d28759b106b2c8aa5ddc" exitCode=137 Oct 30 19:30:30.910281 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:30.910124 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7106-c3b1dd" Oct 30 19:30:30.910281 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:30.910173 4531 scope.go:122] "RemoveContainer" containerID="0c932f2ddfbc1ffcc0a805142b30e3ebe0b7a7caf228d28759b106b2c8aa5ddc" Oct 30 19:30:30.910464 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:30.910439 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7106-9d0e92" podUID="a6acdaee-969b-4087-9563-4ef8f8274df8" containerName="test-curl-7106-9d0e92" containerID="cri-o://7fb7f4efcb8729a71d30c9ee579e6c8343ab65eed82c5b7c2c322c5c239fb0f4" gracePeriod=2 Oct 30 19:30:30.919982 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:30.919962 4531 scope.go:122] "RemoveContainer" containerID="0c932f2ddfbc1ffcc0a805142b30e3ebe0b7a7caf228d28759b106b2c8aa5ddc" Oct 30 19:30:30.920483 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:30.920463 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c932f2ddfbc1ffcc0a805142b30e3ebe0b7a7caf228d28759b106b2c8aa5ddc\": container with ID starting with 0c932f2ddfbc1ffcc0a805142b30e3ebe0b7a7caf228d28759b106b2c8aa5ddc not found: ID does not exist" containerID="0c932f2ddfbc1ffcc0a805142b30e3ebe0b7a7caf228d28759b106b2c8aa5ddc" Oct 30 19:30:30.920554 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:30.920494 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c932f2ddfbc1ffcc0a805142b30e3ebe0b7a7caf228d28759b106b2c8aa5ddc"} err="failed to get container status \"0c932f2ddfbc1ffcc0a805142b30e3ebe0b7a7caf228d28759b106b2c8aa5ddc\": rpc error: code = NotFound desc = could not find container \"0c932f2ddfbc1ffcc0a805142b30e3ebe0b7a7caf228d28759b106b2c8aa5ddc\": container with ID starting with 0c932f2ddfbc1ffcc0a805142b30e3ebe0b7a7caf228d28759b106b2c8aa5ddc not found: ID does not exist" Oct 30 19:30:30.939477 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:30.939435 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbfmz\" (UniqueName: \"kubernetes.io/projected/316519ba-9ca9-42f9-9bdd-659fe52ee33c-kube-api-access-vbfmz\") pod \"test-curl-7106-749cf7\" (UID: \"316519ba-9ca9-42f9-9bdd-659fe52ee33c\") " pod="opendatahub/test-curl-7106-749cf7" Oct 30 19:30:30.961174 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:30.961121 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-c3b1dd\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="022277ec-105e-4bec-97d5-d2de22ec5ddc" pod="opendatahub/test-curl-7106-c3b1dd" Oct 30 19:30:31.000848 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:31.000812 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbfmz\" (UniqueName: \"kubernetes.io/projected/316519ba-9ca9-42f9-9bdd-659fe52ee33c-kube-api-access-vbfmz\") pod \"test-curl-7106-749cf7\" (UID: \"316519ba-9ca9-42f9-9bdd-659fe52ee33c\") " pod="opendatahub/test-curl-7106-749cf7" Oct 30 19:30:31.011908 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:31.011856 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-9d0e92\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="a6acdaee-969b-4087-9563-4ef8f8274df8" pod="opendatahub/test-curl-7106-9d0e92" Oct 30 19:30:31.048351 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:31.048303 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7106-749cf7" Oct 30 19:30:31.062750 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:31.062700 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-c3b1dd\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="022277ec-105e-4bec-97d5-d2de22ec5ddc" pod="opendatahub/test-curl-7106-c3b1dd" Oct 30 19:30:31.433768 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:31.433640 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7106-749cf7"] Oct 30 19:30:31.436678 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:30:31.436645 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod316519ba_9ca9_42f9_9bdd_659fe52ee33c.slice/crio-d4f67e2756ab129fc0a8b5a79de291676d080417451f7480bddd59740c75c06f WatchSource:0}: Error finding container d4f67e2756ab129fc0a8b5a79de291676d080417451f7480bddd59740c75c06f: Status 404 returned error can't find the container with id d4f67e2756ab129fc0a8b5a79de291676d080417451f7480bddd59740c75c06f Oct 30 19:30:31.922044 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:31.921515 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7106-749cf7" event={"ID":"316519ba-9ca9-42f9-9bdd-659fe52ee33c","Type":"ContainerStarted","Data":"b99343241c1a8c677067240bef615b49309145a6025d6cbfb5774f088cbdafd7"} Oct 30 19:30:31.922044 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:31.921581 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7106-749cf7" event={"ID":"316519ba-9ca9-42f9-9bdd-659fe52ee33c","Type":"ContainerStarted","Data":"d4f67e2756ab129fc0a8b5a79de291676d080417451f7480bddd59740c75c06f"} Oct 30 19:30:32.040447 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:32.040378 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7106-749cf7" podStartSLOduration=2.040361292 podStartE2EDuration="2.040361292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:32.037631158 +0000 UTC m=+3392.064736117" watchObservedRunningTime="2026-10-30 19:30:32.040361292 +0000 UTC m=+3392.067466252" Oct 30 19:30:32.088854 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:32.088797 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-9d0e92\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="a6acdaee-969b-4087-9563-4ef8f8274df8" pod="opendatahub/test-curl-7106-9d0e92" Oct 30 19:30:32.100657 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:32.100618 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="022277ec-105e-4bec-97d5-d2de22ec5ddc" path="/var/lib/kubelet/pods/022277ec-105e-4bec-97d5-d2de22ec5ddc/volumes" Oct 30 19:30:32.144558 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:32.144508 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-c3b1dd\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="022277ec-105e-4bec-97d5-d2de22ec5ddc" pod="opendatahub/test-curl-7106-c3b1dd" Oct 30 19:30:33.166026 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:33.165946 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7106-9d0e92" Oct 30 19:30:33.264769 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:33.264732 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6acdaee-969b-4087-9563-4ef8f8274df8-kube-api-access-nlzzp\" (UniqueName: \"kubernetes.io/projected/a6acdaee-969b-4087-9563-4ef8f8274df8-kube-api-access-nlzzp\") pod \"a6acdaee-969b-4087-9563-4ef8f8274df8\" (UID: \"a6acdaee-969b-4087-9563-4ef8f8274df8\") " Oct 30 19:30:33.267112 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:33.267079 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6acdaee-969b-4087-9563-4ef8f8274df8-kube-api-access-nlzzp" pod "a6acdaee-969b-4087-9563-4ef8f8274df8" (UID: "a6acdaee-969b-4087-9563-4ef8f8274df8"). InnerVolumeSpecName "kube-api-access-nlzzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:33.366544 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:33.366484 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlzzp\" (UniqueName: \"kubernetes.io/projected/a6acdaee-969b-4087-9563-4ef8f8274df8-kube-api-access-nlzzp\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:33.527935 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:33.527890 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7106-749cf7"] Oct 30 19:30:33.533540 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:33.533495 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7106-749cf7"] Oct 30 19:30:33.708205 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:33.708151 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7106-a89f59"] Oct 30 19:30:33.708895 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:33.708873 4531 cpu_manager.go:414] "containerMap: removing container" podUID="316519ba-9ca9-42f9-9bdd-659fe52ee33c" containerName="test-curl-7106-749cf7" Oct 30 19:30:33.709004 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:33.708897 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="316519ba-9ca9-42f9-9bdd-659fe52ee33c" containerName="test-curl-7106-749cf7" Oct 30 19:30:33.709080 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:33.709007 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="316519ba-9ca9-42f9-9bdd-659fe52ee33c" containerName="test-curl-7106-749cf7" Oct 30 19:30:33.712567 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:33.712540 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7106-a89f59" Oct 30 19:30:33.771145 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:33.771079 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzt56\" (UniqueName: \"kubernetes.io/projected/a3503e36-1b73-4ef6-b12b-7c8a93ef6285-kube-api-access-kzt56\") pod \"test-curl-7106-a89f59\" (UID: \"a3503e36-1b73-4ef6-b12b-7c8a93ef6285\") " pod="opendatahub/test-curl-7106-a89f59" Oct 30 19:30:33.792529 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:33.792426 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7106-a89f59"] Oct 30 19:30:33.872667 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:33.872621 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzt56\" (UniqueName: \"kubernetes.io/projected/a3503e36-1b73-4ef6-b12b-7c8a93ef6285-kube-api-access-kzt56\") pod \"test-curl-7106-a89f59\" (UID: \"a3503e36-1b73-4ef6-b12b-7c8a93ef6285\") " pod="opendatahub/test-curl-7106-a89f59" Oct 30 19:30:33.934353 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:33.934315 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzt56\" (UniqueName: \"kubernetes.io/projected/a3503e36-1b73-4ef6-b12b-7c8a93ef6285-kube-api-access-kzt56\") pod \"test-curl-7106-a89f59\" (UID: \"a3503e36-1b73-4ef6-b12b-7c8a93ef6285\") " pod="opendatahub/test-curl-7106-a89f59" Oct 30 19:30:33.941727 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:33.941676 4531 generic.go:356] "Generic (PLEG): container finished" podID="a6acdaee-969b-4087-9563-4ef8f8274df8" containerID="7fb7f4efcb8729a71d30c9ee579e6c8343ab65eed82c5b7c2c322c5c239fb0f4" exitCode=137 Oct 30 19:30:33.941926 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:33.941786 4531 scope.go:122] "RemoveContainer" containerID="7fb7f4efcb8729a71d30c9ee579e6c8343ab65eed82c5b7c2c322c5c239fb0f4" Oct 30 19:30:33.941926 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:33.941867 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7106-9d0e92" Oct 30 19:30:33.942024 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:33.941919 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7106-749cf7" podUID="316519ba-9ca9-42f9-9bdd-659fe52ee33c" containerName="test-curl-7106-749cf7" containerID="cri-o://b99343241c1a8c677067240bef615b49309145a6025d6cbfb5774f088cbdafd7" gracePeriod=2 Oct 30 19:30:33.981351 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:33.981310 4531 scope.go:122] "RemoveContainer" containerID="7fb7f4efcb8729a71d30c9ee579e6c8343ab65eed82c5b7c2c322c5c239fb0f4" Oct 30 19:30:33.981873 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:33.981840 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb7f4efcb8729a71d30c9ee579e6c8343ab65eed82c5b7c2c322c5c239fb0f4\": container with ID starting with 7fb7f4efcb8729a71d30c9ee579e6c8343ab65eed82c5b7c2c322c5c239fb0f4 not found: ID does not exist" containerID="7fb7f4efcb8729a71d30c9ee579e6c8343ab65eed82c5b7c2c322c5c239fb0f4" Oct 30 19:30:33.981996 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:33.981890 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb7f4efcb8729a71d30c9ee579e6c8343ab65eed82c5b7c2c322c5c239fb0f4"} err="failed to get container status \"7fb7f4efcb8729a71d30c9ee579e6c8343ab65eed82c5b7c2c322c5c239fb0f4\": rpc error: code = NotFound desc = could not find container \"7fb7f4efcb8729a71d30c9ee579e6c8343ab65eed82c5b7c2c322c5c239fb0f4\": container with ID starting with 7fb7f4efcb8729a71d30c9ee579e6c8343ab65eed82c5b7c2c322c5c239fb0f4 not found: ID does not exist" Oct 30 19:30:33.993837 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:33.993599 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-749cf7\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="316519ba-9ca9-42f9-9bdd-659fe52ee33c" pod="opendatahub/test-curl-7106-749cf7" Oct 30 19:30:34.025547 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:34.025492 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7106-a89f59" Oct 30 19:30:34.044741 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:34.044690 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-749cf7\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="316519ba-9ca9-42f9-9bdd-659fe52ee33c" pod="opendatahub/test-curl-7106-749cf7" Oct 30 19:30:34.095754 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:34.095707 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-9d0e92\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="a6acdaee-969b-4087-9563-4ef8f8274df8" pod="opendatahub/test-curl-7106-9d0e92" Oct 30 19:30:34.115929 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:34.115343 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6acdaee-969b-4087-9563-4ef8f8274df8" path="/var/lib/kubelet/pods/a6acdaee-969b-4087-9563-4ef8f8274df8/volumes" Oct 30 19:30:34.253245 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:34.253208 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a2e084f-bbs2x"] Oct 30 19:30:34.258728 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:34.258697 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2e084f-bbs2x" Oct 30 19:30:34.316437 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:34.316336 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a2e084f-bbs2x"] Oct 30 19:30:34.359998 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:34.359965 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:30:34.359998 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:34.359994 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:30:34.360269 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:34.360023 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:30:34.361990 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:34.361971 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:30:34.387097 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:34.387062 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cf097bb-7649-46a5-a1a7-ba3359454574-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a2e084f-bbs2x\" (UID: \"6cf097bb-7649-46a5-a1a7-ba3359454574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2e084f-bbs2x" Oct 30 19:30:34.387320 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:34.387148 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cf097bb-7649-46a5-a1a7-ba3359454574-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a2e084f-bbs2x\" (UID: \"6cf097bb-7649-46a5-a1a7-ba3359454574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2e084f-bbs2x" Oct 30 19:30:34.387368 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:34.387317 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmdvr\" (UniqueName: \"kubernetes.io/projected/6cf097bb-7649-46a5-a1a7-ba3359454574-kube-api-access-fmdvr\") pod \"maas-api-revoke-sub-7a2e084f-bbs2x\" (UID: \"6cf097bb-7649-46a5-a1a7-ba3359454574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2e084f-bbs2x" Oct 30 19:30:34.421089 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:34.421063 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7106-a89f59"] Oct 30 19:30:34.423856 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:30:34.423821 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3503e36_1b73_4ef6_b12b_7c8a93ef6285.slice/crio-93a9d84de8ed8d31c9223661fadc121fe206a49cafba564c43d372b4013a3606 WatchSource:0}: Error finding container 93a9d84de8ed8d31c9223661fadc121fe206a49cafba564c43d372b4013a3606: Status 404 returned error can't find the container with id 93a9d84de8ed8d31c9223661fadc121fe206a49cafba564c43d372b4013a3606 Oct 30 19:30:34.488692 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:34.488657 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cf097bb-7649-46a5-a1a7-ba3359454574-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a2e084f-bbs2x\" (UID: \"6cf097bb-7649-46a5-a1a7-ba3359454574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2e084f-bbs2x" Oct 30 19:30:34.488867 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:34.488797 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmdvr\" (UniqueName: \"kubernetes.io/projected/6cf097bb-7649-46a5-a1a7-ba3359454574-kube-api-access-fmdvr\") pod \"maas-api-revoke-sub-7a2e084f-bbs2x\" (UID: \"6cf097bb-7649-46a5-a1a7-ba3359454574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2e084f-bbs2x" Oct 30 19:30:34.488944 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:34.488897 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cf097bb-7649-46a5-a1a7-ba3359454574-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a2e084f-bbs2x\" (UID: \"6cf097bb-7649-46a5-a1a7-ba3359454574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2e084f-bbs2x" Oct 30 19:30:34.489429 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:34.489404 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cf097bb-7649-46a5-a1a7-ba3359454574-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a2e084f-bbs2x\" (UID: \"6cf097bb-7649-46a5-a1a7-ba3359454574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2e084f-bbs2x" Oct 30 19:30:34.491432 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:34.491405 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cf097bb-7649-46a5-a1a7-ba3359454574-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a2e084f-bbs2x\" (UID: \"6cf097bb-7649-46a5-a1a7-ba3359454574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2e084f-bbs2x" Oct 30 19:30:34.550414 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:34.550376 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmdvr\" (UniqueName: \"kubernetes.io/projected/6cf097bb-7649-46a5-a1a7-ba3359454574-kube-api-access-fmdvr\") pod \"maas-api-revoke-sub-7a2e084f-bbs2x\" (UID: \"6cf097bb-7649-46a5-a1a7-ba3359454574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2e084f-bbs2x" Oct 30 19:30:34.575346 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:34.575252 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2e084f-bbs2x" Oct 30 19:30:34.951001 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:34.950918 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7106-a89f59" event={"ID":"a3503e36-1b73-4ef6-b12b-7c8a93ef6285","Type":"ContainerStarted","Data":"d9290686577d0db181f1ca4b15cf6e5c81e3399ac83758937c05988793afd161"} Oct 30 19:30:34.951001 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:34.950968 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7106-a89f59" event={"ID":"a3503e36-1b73-4ef6-b12b-7c8a93ef6285","Type":"ContainerStarted","Data":"93a9d84de8ed8d31c9223661fadc121fe206a49cafba564c43d372b4013a3606"} Oct 30 19:30:34.960743 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:34.960716 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a2e084f-bbs2x"] Oct 30 19:30:34.962816 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:30:34.962778 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cf097bb_7649_46a5_a1a7_ba3359454574.slice/crio-202e74d4135a0123493adeda5caf470acba550d3001ef25bb874a5c7d8f4bdd3 WatchSource:0}: Error finding container 202e74d4135a0123493adeda5caf470acba550d3001ef25bb874a5c7d8f4bdd3: Status 404 returned error can't find the container with id 202e74d4135a0123493adeda5caf470acba550d3001ef25bb874a5c7d8f4bdd3 Oct 30 19:30:35.070360 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:35.070303 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7106-a89f59" podStartSLOduration=2.070284293 podStartE2EDuration="2.070284293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:35.069418766 +0000 UTC m=+3395.096523725" watchObservedRunningTime="2026-10-30 19:30:35.070284293 +0000 UTC m=+3395.097389253" Oct 30 19:30:35.959591 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:35.959549 4531 generic.go:356] "Generic (PLEG): container finished" podID="6cf097bb-7649-46a5-a1a7-ba3359454574" containerID="d6f4639d2ec4cf8bb01a1609e00040eebb92b92ef6c12d1d9831605b3e5efcdb" exitCode=0 Oct 30 19:30:35.960000 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:35.959862 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2e084f-bbs2x" event={"ID":"6cf097bb-7649-46a5-a1a7-ba3359454574","Type":"ContainerDied","Data":"d6f4639d2ec4cf8bb01a1609e00040eebb92b92ef6c12d1d9831605b3e5efcdb"} Oct 30 19:30:35.960000 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:35.959905 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2e084f-bbs2x" event={"ID":"6cf097bb-7649-46a5-a1a7-ba3359454574","Type":"ContainerStarted","Data":"202e74d4135a0123493adeda5caf470acba550d3001ef25bb874a5c7d8f4bdd3"} Oct 30 19:30:36.163362 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:36.163329 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7106-749cf7" Oct 30 19:30:36.209879 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:36.209770 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/316519ba-9ca9-42f9-9bdd-659fe52ee33c-kube-api-access-vbfmz\" (UniqueName: \"kubernetes.io/projected/316519ba-9ca9-42f9-9bdd-659fe52ee33c-kube-api-access-vbfmz\") pod \"316519ba-9ca9-42f9-9bdd-659fe52ee33c\" (UID: \"316519ba-9ca9-42f9-9bdd-659fe52ee33c\") " Oct 30 19:30:36.212346 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:36.212306 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/316519ba-9ca9-42f9-9bdd-659fe52ee33c-kube-api-access-vbfmz" pod "316519ba-9ca9-42f9-9bdd-659fe52ee33c" (UID: "316519ba-9ca9-42f9-9bdd-659fe52ee33c"). InnerVolumeSpecName "kube-api-access-vbfmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:36.311441 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:36.311397 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbfmz\" (UniqueName: \"kubernetes.io/projected/316519ba-9ca9-42f9-9bdd-659fe52ee33c-kube-api-access-vbfmz\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:36.606644 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:36.606602 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7106-a89f59"] Oct 30 19:30:36.617239 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:36.617194 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7106-a89f59"] Oct 30 19:30:36.968006 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:36.967903 4531 generic.go:356] "Generic (PLEG): container finished" podID="316519ba-9ca9-42f9-9bdd-659fe52ee33c" containerID="b99343241c1a8c677067240bef615b49309145a6025d6cbfb5774f088cbdafd7" exitCode=137 Oct 30 19:30:36.968006 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:36.967953 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7106-749cf7" Oct 30 19:30:36.968006 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:36.968003 4531 scope.go:122] "RemoveContainer" containerID="b99343241c1a8c677067240bef615b49309145a6025d6cbfb5774f088cbdafd7" Oct 30 19:30:36.968656 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:36.968413 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7106-a89f59" podUID="a3503e36-1b73-4ef6-b12b-7c8a93ef6285" containerName="test-curl-7106-a89f59" containerID="cri-o://d9290686577d0db181f1ca4b15cf6e5c81e3399ac83758937c05988793afd161" gracePeriod=2 Oct 30 19:30:36.973565 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:36.973518 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" podUID="cd2ab45c-944c-4d14-8362-0831d16a2c59" containerName="main" containerID="cri-o://a2988b915d0f67cb24083a09774c47df58fc2aa49ee3776fd45033b8345ffdca" gracePeriod=45 Oct 30 19:30:36.980653 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:36.980630 4531 scope.go:122] "RemoveContainer" containerID="b99343241c1a8c677067240bef615b49309145a6025d6cbfb5774f088cbdafd7" Oct 30 19:30:36.980988 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:36.980963 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b99343241c1a8c677067240bef615b49309145a6025d6cbfb5774f088cbdafd7\": container with ID starting with b99343241c1a8c677067240bef615b49309145a6025d6cbfb5774f088cbdafd7 not found: ID does not exist" containerID="b99343241c1a8c677067240bef615b49309145a6025d6cbfb5774f088cbdafd7" Oct 30 19:30:36.981060 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:36.981004 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b99343241c1a8c677067240bef615b49309145a6025d6cbfb5774f088cbdafd7"} err="failed to get container status \"b99343241c1a8c677067240bef615b49309145a6025d6cbfb5774f088cbdafd7\": rpc error: code = NotFound desc = could not find container \"b99343241c1a8c677067240bef615b49309145a6025d6cbfb5774f088cbdafd7\": container with ID starting with b99343241c1a8c677067240bef615b49309145a6025d6cbfb5774f088cbdafd7 not found: ID does not exist" Oct 30 19:30:37.019398 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:37.019356 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-a89f59\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="a3503e36-1b73-4ef6-b12b-7c8a93ef6285" pod="opendatahub/test-curl-7106-a89f59" Oct 30 19:30:37.070279 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:37.070237 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-749cf7\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="316519ba-9ca9-42f9-9bdd-659fe52ee33c" pod="opendatahub/test-curl-7106-749cf7" Oct 30 19:30:37.121183 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:37.121097 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-a89f59\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="a3503e36-1b73-4ef6-b12b-7c8a93ef6285" pod="opendatahub/test-curl-7106-a89f59" Oct 30 19:30:37.138789 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.138754 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2e084f-bbs2x" Oct 30 19:30:37.190050 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:37.189995 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-749cf7\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="316519ba-9ca9-42f9-9bdd-659fe52ee33c" pod="opendatahub/test-curl-7106-749cf7" Oct 30 19:30:37.221916 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.221823 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cf097bb-7649-46a5-a1a7-ba3359454574-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cf097bb-7649-46a5-a1a7-ba3359454574-maas-api-xks-ca\") pod \"6cf097bb-7649-46a5-a1a7-ba3359454574\" (UID: \"6cf097bb-7649-46a5-a1a7-ba3359454574\") " Oct 30 19:30:37.222115 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.221963 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cf097bb-7649-46a5-a1a7-ba3359454574-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cf097bb-7649-46a5-a1a7-ba3359454574-maas-api-openshift-ca\") pod \"6cf097bb-7649-46a5-a1a7-ba3359454574\" (UID: \"6cf097bb-7649-46a5-a1a7-ba3359454574\") " Oct 30 19:30:37.222115 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.222002 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cf097bb-7649-46a5-a1a7-ba3359454574-kube-api-access-fmdvr\" (UniqueName: \"kubernetes.io/projected/6cf097bb-7649-46a5-a1a7-ba3359454574-kube-api-access-fmdvr\") pod \"6cf097bb-7649-46a5-a1a7-ba3359454574\" (UID: \"6cf097bb-7649-46a5-a1a7-ba3359454574\") " Oct 30 19:30:37.222608 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.222393 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cf097bb-7649-46a5-a1a7-ba3359454574-maas-api-openshift-ca" pod "6cf097bb-7649-46a5-a1a7-ba3359454574" (UID: "6cf097bb-7649-46a5-a1a7-ba3359454574"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:37.222818 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.222795 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cf097bb-7649-46a5-a1a7-ba3359454574-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:37.224318 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.224286 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cf097bb-7649-46a5-a1a7-ba3359454574-maas-api-xks-ca" pod "6cf097bb-7649-46a5-a1a7-ba3359454574" (UID: "6cf097bb-7649-46a5-a1a7-ba3359454574"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:37.224559 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.224536 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cf097bb-7649-46a5-a1a7-ba3359454574-kube-api-access-fmdvr" pod "6cf097bb-7649-46a5-a1a7-ba3359454574" (UID: "6cf097bb-7649-46a5-a1a7-ba3359454574"). InnerVolumeSpecName "kube-api-access-fmdvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:37.243337 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:37.243287 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-a89f59\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="a3503e36-1b73-4ef6-b12b-7c8a93ef6285" pod="opendatahub/test-curl-7106-a89f59" Oct 30 19:30:37.253902 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.253845 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" Oct 30 19:30:37.323554 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.323509 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-home\" (UniqueName: \"kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-home\") pod \"cd2ab45c-944c-4d14-8362-0831d16a2c59\" (UID: \"cd2ab45c-944c-4d14-8362-0831d16a2c59\") " Oct 30 19:30:37.323554 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.323555 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-tmp-dir\") pod \"cd2ab45c-944c-4d14-8362-0831d16a2c59\" (UID: \"cd2ab45c-944c-4d14-8362-0831d16a2c59\") " Oct 30 19:30:37.323831 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.323634 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-dshm\") pod \"cd2ab45c-944c-4d14-8362-0831d16a2c59\" (UID: \"cd2ab45c-944c-4d14-8362-0831d16a2c59\") " Oct 30 19:30:37.323831 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.323693 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd2ab45c-944c-4d14-8362-0831d16a2c59-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd2ab45c-944c-4d14-8362-0831d16a2c59-tls-certs\") pod \"cd2ab45c-944c-4d14-8362-0831d16a2c59\" (UID: \"cd2ab45c-944c-4d14-8362-0831d16a2c59\") " Oct 30 19:30:37.323831 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.323806 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd2ab45c-944c-4d14-8362-0831d16a2c59-kube-api-access-bmmsl\" (UniqueName: \"kubernetes.io/projected/cd2ab45c-944c-4d14-8362-0831d16a2c59-kube-api-access-bmmsl\") pod \"cd2ab45c-944c-4d14-8362-0831d16a2c59\" (UID: \"cd2ab45c-944c-4d14-8362-0831d16a2c59\") " Oct 30 19:30:37.324012 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.323831 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-model-cache\") pod \"cd2ab45c-944c-4d14-8362-0831d16a2c59\" (UID: \"cd2ab45c-944c-4d14-8362-0831d16a2c59\") " Oct 30 19:30:37.324012 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.323874 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-tmp-dir" pod "cd2ab45c-944c-4d14-8362-0831d16a2c59" (UID: "cd2ab45c-944c-4d14-8362-0831d16a2c59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:30:37.324012 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.323921 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-home" pod "cd2ab45c-944c-4d14-8362-0831d16a2c59" (UID: "cd2ab45c-944c-4d14-8362-0831d16a2c59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:30:37.324261 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.324207 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmdvr\" (UniqueName: \"kubernetes.io/projected/6cf097bb-7649-46a5-a1a7-ba3359454574-kube-api-access-fmdvr\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:37.324261 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.324227 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cf097bb-7649-46a5-a1a7-ba3359454574-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:37.324261 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.324241 4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-home\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:37.324261 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.324257 4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-tmp-dir\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:37.324507 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.324481 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-model-cache" pod "cd2ab45c-944c-4d14-8362-0831d16a2c59" (UID: "cd2ab45c-944c-4d14-8362-0831d16a2c59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:30:37.326871 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.326823 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd2ab45c-944c-4d14-8362-0831d16a2c59-kube-api-access-bmmsl" pod "cd2ab45c-944c-4d14-8362-0831d16a2c59" (UID: "cd2ab45c-944c-4d14-8362-0831d16a2c59"). InnerVolumeSpecName "kube-api-access-bmmsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:37.327060 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.326869 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd2ab45c-944c-4d14-8362-0831d16a2c59-tls-certs" pod "cd2ab45c-944c-4d14-8362-0831d16a2c59" (UID: "cd2ab45c-944c-4d14-8362-0831d16a2c59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:37.327060 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.326936 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-dshm" pod "cd2ab45c-944c-4d14-8362-0831d16a2c59" (UID: "cd2ab45c-944c-4d14-8362-0831d16a2c59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:30:37.425841 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.425795 4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-dshm\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:37.425841 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.425830 4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd2ab45c-944c-4d14-8362-0831d16a2c59-tls-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:37.425841 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.425840 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmmsl\" (UniqueName: \"kubernetes.io/projected/cd2ab45c-944c-4d14-8362-0831d16a2c59-kube-api-access-bmmsl\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:37.425841 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.425851 4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd2ab45c-944c-4d14-8362-0831d16a2c59-model-cache\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:37.528094 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:37.528051 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-749cf7\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="316519ba-9ca9-42f9-9bdd-659fe52ee33c" pod="opendatahub/test-curl-7106-749cf7" Oct 30 19:30:37.578822 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:37.578781 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-a89f59\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="a3503e36-1b73-4ef6-b12b-7c8a93ef6285" pod="opendatahub/test-curl-7106-a89f59" Oct 30 19:30:37.977556 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.977443 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2e084f-bbs2x" Oct 30 19:30:37.977556 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.977493 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2e084f-bbs2x" event={"ID":"6cf097bb-7649-46a5-a1a7-ba3359454574","Type":"ContainerDied","Data":"202e74d4135a0123493adeda5caf470acba550d3001ef25bb874a5c7d8f4bdd3"} Oct 30 19:30:37.978191 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.977575 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="202e74d4135a0123493adeda5caf470acba550d3001ef25bb874a5c7d8f4bdd3" Oct 30 19:30:37.979069 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.979043 4531 generic.go:356] "Generic (PLEG): container finished" podID="cd2ab45c-944c-4d14-8362-0831d16a2c59" containerID="a2988b915d0f67cb24083a09774c47df58fc2aa49ee3776fd45033b8345ffdca" exitCode=0 Oct 30 19:30:37.979227 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.979087 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" event={"ID":"cd2ab45c-944c-4d14-8362-0831d16a2c59","Type":"ContainerDied","Data":"a2988b915d0f67cb24083a09774c47df58fc2aa49ee3776fd45033b8345ffdca"} Oct 30 19:30:37.979227 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.979113 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" event={"ID":"cd2ab45c-944c-4d14-8362-0831d16a2c59","Type":"ContainerDied","Data":"6ce26e3b3b4b23987fc48ba54665bca17aefe421401bfad40d5821dfad8fdb38"} Oct 30 19:30:37.979227 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.979159 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x" Oct 30 19:30:37.979227 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:37.979169 4531 scope.go:122] "RemoveContainer" containerID="a2988b915d0f67cb24083a09774c47df58fc2aa49ee3776fd45033b8345ffdca" Oct 30 19:30:38.000399 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:38.000374 4531 scope.go:122] "RemoveContainer" containerID="a2988b915d0f67cb24083a09774c47df58fc2aa49ee3776fd45033b8345ffdca" Oct 30 19:30:38.000821 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:38.000798 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2988b915d0f67cb24083a09774c47df58fc2aa49ee3776fd45033b8345ffdca\": container with ID starting with a2988b915d0f67cb24083a09774c47df58fc2aa49ee3776fd45033b8345ffdca not found: ID does not exist" containerID="a2988b915d0f67cb24083a09774c47df58fc2aa49ee3776fd45033b8345ffdca" Oct 30 19:30:38.000941 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:38.000831 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2988b915d0f67cb24083a09774c47df58fc2aa49ee3776fd45033b8345ffdca"} err="failed to get container status \"a2988b915d0f67cb24083a09774c47df58fc2aa49ee3776fd45033b8345ffdca\": rpc error: code = NotFound desc = could not find container \"a2988b915d0f67cb24083a09774c47df58fc2aa49ee3776fd45033b8345ffdca\": container with ID starting with a2988b915d0f67cb24083a09774c47df58fc2aa49ee3776fd45033b8345ffdca not found: ID does not exist" Oct 30 19:30:38.082581 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:38.082536 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-749cf7\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="316519ba-9ca9-42f9-9bdd-659fe52ee33c" pod="opendatahub/test-curl-7106-749cf7" Oct 30 19:30:38.099387 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:38.099349 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="316519ba-9ca9-42f9-9bdd-659fe52ee33c" path="/var/lib/kubelet/pods/316519ba-9ca9-42f9-9bdd-659fe52ee33c/volumes" Oct 30 19:30:38.133310 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:38.133263 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-a89f59\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="a3503e36-1b73-4ef6-b12b-7c8a93ef6285" pod="opendatahub/test-curl-7106-a89f59" Oct 30 19:30:38.251208 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:38.251167 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x"] Oct 30 19:30:38.255854 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:38.255814 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-8f9c6b/auth-test-model-8f9c6b-kserve-56bf6446c7-zlj9x"] Oct 30 19:30:39.210121 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:39.210088 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7106-a89f59" Oct 30 19:30:39.359742 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:39.359698 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3503e36-1b73-4ef6-b12b-7c8a93ef6285-kube-api-access-kzt56\" (UniqueName: \"kubernetes.io/projected/a3503e36-1b73-4ef6-b12b-7c8a93ef6285-kube-api-access-kzt56\") pod \"a3503e36-1b73-4ef6-b12b-7c8a93ef6285\" (UID: \"a3503e36-1b73-4ef6-b12b-7c8a93ef6285\") " Oct 30 19:30:39.362189 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:39.362153 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3503e36-1b73-4ef6-b12b-7c8a93ef6285-kube-api-access-kzt56" pod "a3503e36-1b73-4ef6-b12b-7c8a93ef6285" (UID: "a3503e36-1b73-4ef6-b12b-7c8a93ef6285"). InnerVolumeSpecName "kube-api-access-kzt56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:39.460807 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:39.460766 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzt56\" (UniqueName: \"kubernetes.io/projected/a3503e36-1b73-4ef6-b12b-7c8a93ef6285-kube-api-access-kzt56\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:39.995769 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:39.995731 4531 generic.go:356] "Generic (PLEG): container finished" podID="a3503e36-1b73-4ef6-b12b-7c8a93ef6285" containerID="d9290686577d0db181f1ca4b15cf6e5c81e3399ac83758937c05988793afd161" exitCode=137 Oct 30 19:30:39.996002 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:39.995779 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7106-a89f59" Oct 30 19:30:39.996002 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:39.995799 4531 scope.go:122] "RemoveContainer" containerID="d9290686577d0db181f1ca4b15cf6e5c81e3399ac83758937c05988793afd161" Oct 30 19:30:40.005801 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:40.005613 4531 scope.go:122] "RemoveContainer" containerID="d9290686577d0db181f1ca4b15cf6e5c81e3399ac83758937c05988793afd161" Oct 30 19:30:40.006064 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:40.006036 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9290686577d0db181f1ca4b15cf6e5c81e3399ac83758937c05988793afd161\": container with ID starting with d9290686577d0db181f1ca4b15cf6e5c81e3399ac83758937c05988793afd161 not found: ID does not exist" containerID="d9290686577d0db181f1ca4b15cf6e5c81e3399ac83758937c05988793afd161" Oct 30 19:30:40.006190 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:40.006078 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9290686577d0db181f1ca4b15cf6e5c81e3399ac83758937c05988793afd161"} err="failed to get container status \"d9290686577d0db181f1ca4b15cf6e5c81e3399ac83758937c05988793afd161\": rpc error: code = NotFound desc = could not find container \"d9290686577d0db181f1ca4b15cf6e5c81e3399ac83758937c05988793afd161\": container with ID starting with d9290686577d0db181f1ca4b15cf6e5c81e3399ac83758937c05988793afd161 not found: ID does not exist" Oct 30 19:30:40.055339 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:40.055301 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-a89f59\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="a3503e36-1b73-4ef6-b12b-7c8a93ef6285" pod="opendatahub/test-curl-7106-a89f59" Oct 30 19:30:40.099201 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:40.099153 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3503e36-1b73-4ef6-b12b-7c8a93ef6285" path="/var/lib/kubelet/pods/a3503e36-1b73-4ef6-b12b-7c8a93ef6285/volumes" Oct 30 19:30:40.099754 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:40.099731 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd2ab45c-944c-4d14-8362-0831d16a2c59" path="/var/lib/kubelet/pods/cd2ab45c-944c-4d14-8362-0831d16a2c59/volumes" Oct 30 19:30:40.146193 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:40.146119 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7106-a89f59\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="a3503e36-1b73-4ef6-b12b-7c8a93ef6285" pod="opendatahub/test-curl-7106-a89f59" Oct 30 19:30:42.436665 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.436624 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v"] Oct 30 19:30:42.437119 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.437066 4531 cpu_manager.go:414] "containerMap: removing container" podUID="cd2ab45c-944c-4d14-8362-0831d16a2c59" containerName="main" Oct 30 19:30:42.437119 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.437078 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd2ab45c-944c-4d14-8362-0831d16a2c59" containerName="main" Oct 30 19:30:42.437119 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.437095 4531 cpu_manager.go:414] "containerMap: removing container" podUID="6cf097bb-7649-46a5-a1a7-ba3359454574" containerName="revoke-keys" Oct 30 19:30:42.437119 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.437100 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cf097bb-7649-46a5-a1a7-ba3359454574" containerName="revoke-keys" Oct 30 19:30:42.437278 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.437141 4531 cpu_manager.go:414] "containerMap: removing container" podUID="a3503e36-1b73-4ef6-b12b-7c8a93ef6285" containerName="test-curl-7106-a89f59" Oct 30 19:30:42.437278 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.437147 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3503e36-1b73-4ef6-b12b-7c8a93ef6285" containerName="test-curl-7106-a89f59" Oct 30 19:30:42.437278 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.437217 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3503e36-1b73-4ef6-b12b-7c8a93ef6285" containerName="test-curl-7106-a89f59" Oct 30 19:30:42.437278 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.437230 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd2ab45c-944c-4d14-8362-0831d16a2c59" containerName="main" Oct 30 19:30:42.437278 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.437237 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cf097bb-7649-46a5-a1a7-ba3359454574" containerName="revoke-keys" Oct 30 19:30:42.441857 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.441832 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v" Oct 30 19:30:42.496575 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.496534 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-da1b8b\"" Oct 30 19:30:42.546187 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.546112 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-da1b8b-dockercfg-trznd\"" Oct 30 19:30:42.549778 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.549742 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v"] Oct 30 19:30:42.595800 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.595759 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d666af3a-0ed2-4a2e-bed8-bde459fbdfd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v\" (UID: \"d666af3a-0ed2-4a2e-bed8-bde459fbdfd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v" Oct 30 19:30:42.596004 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.595923 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szbx7\" (UniqueName: \"kubernetes.io/projected/d666af3a-0ed2-4a2e-bed8-bde459fbdfd7-kube-api-access-szbx7\") pod \"payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v\" (UID: \"d666af3a-0ed2-4a2e-bed8-bde459fbdfd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v" Oct 30 19:30:42.619831 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.619785 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg"] Oct 30 19:30:42.623632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.623603 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg" Oct 30 19:30:42.682918 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.682881 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg"] Oct 30 19:30:42.697042 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.696930 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szbx7\" (UniqueName: \"kubernetes.io/projected/d666af3a-0ed2-4a2e-bed8-bde459fbdfd7-kube-api-access-szbx7\") pod \"payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v\" (UID: \"d666af3a-0ed2-4a2e-bed8-bde459fbdfd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v" Oct 30 19:30:42.697240 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.697060 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb7cb\" (UniqueName: \"kubernetes.io/projected/058ae453-9519-4149-80c9-8c8200d7b9f6-kube-api-access-tb7cb\") pod \"payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg\" (UID: \"058ae453-9519-4149-80c9-8c8200d7b9f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg" Oct 30 19:30:42.697240 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.697144 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d666af3a-0ed2-4a2e-bed8-bde459fbdfd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v\" (UID: \"d666af3a-0ed2-4a2e-bed8-bde459fbdfd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v" Oct 30 19:30:42.697240 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.697206 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058ae453-9519-4149-80c9-8c8200d7b9f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg\" (UID: \"058ae453-9519-4149-80c9-8c8200d7b9f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg" Oct 30 19:30:42.698339 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.698310 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d666af3a-0ed2-4a2e-bed8-bde459fbdfd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v\" (UID: \"d666af3a-0ed2-4a2e-bed8-bde459fbdfd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v" Oct 30 19:30:42.758441 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.758406 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szbx7\" (UniqueName: \"kubernetes.io/projected/d666af3a-0ed2-4a2e-bed8-bde459fbdfd7-kube-api-access-szbx7\") pod \"payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v\" (UID: \"d666af3a-0ed2-4a2e-bed8-bde459fbdfd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v" Oct 30 19:30:42.798710 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.798662 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb7cb\" (UniqueName: \"kubernetes.io/projected/058ae453-9519-4149-80c9-8c8200d7b9f6-kube-api-access-tb7cb\") pod \"payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg\" (UID: \"058ae453-9519-4149-80c9-8c8200d7b9f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg" Oct 30 19:30:42.798887 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.798735 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058ae453-9519-4149-80c9-8c8200d7b9f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg\" (UID: \"058ae453-9519-4149-80c9-8c8200d7b9f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg" Oct 30 19:30:42.799460 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.799440 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058ae453-9519-4149-80c9-8c8200d7b9f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg\" (UID: \"058ae453-9519-4149-80c9-8c8200d7b9f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg" Oct 30 19:30:42.860027 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.859989 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb7cb\" (UniqueName: \"kubernetes.io/projected/058ae453-9519-4149-80c9-8c8200d7b9f6-kube-api-access-tb7cb\") pod \"payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg\" (UID: \"058ae453-9519-4149-80c9-8c8200d7b9f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg" Oct 30 19:30:42.942988 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:42.942950 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg" Oct 30 19:30:43.055664 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:43.055628 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v" Oct 30 19:30:43.325553 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:43.325525 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg"] Oct 30 19:30:43.327682 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:30:43.327652 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod058ae453_9519_4149_80c9_8c8200d7b9f6.slice/crio-459e46c4afd87aeb82f9693b78a693e93e94e1b01772b7d75c934030c6cc676b WatchSource:0}: Error finding container 459e46c4afd87aeb82f9693b78a693e93e94e1b01772b7d75c934030c6cc676b: Status 404 returned error can't find the container with id 459e46c4afd87aeb82f9693b78a693e93e94e1b01772b7d75c934030c6cc676b Oct 30 19:30:43.442803 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:43.442772 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v"] Oct 30 19:30:43.444875 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:30:43.444842 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd666af3a_0ed2_4a2e_bed8_bde459fbdfd7.slice/crio-322daddfba962a4dacbd63f845e1b33b0ee7fb188167b035591f12c60a5f2db3 WatchSource:0}: Error finding container 322daddfba962a4dacbd63f845e1b33b0ee7fb188167b035591f12c60a5f2db3: Status 404 returned error can't find the container with id 322daddfba962a4dacbd63f845e1b33b0ee7fb188167b035591f12c60a5f2db3 Oct 30 19:30:44.033096 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:44.033055 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v" event={"ID":"d666af3a-0ed2-4a2e-bed8-bde459fbdfd7","Type":"ContainerStarted","Data":"b5069e91edd9e97e86779599269cb6192a04378b72c74486b106d20b0969f374"} Oct 30 19:30:44.033425 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:44.033116 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v" event={"ID":"d666af3a-0ed2-4a2e-bed8-bde459fbdfd7","Type":"ContainerStarted","Data":"322daddfba962a4dacbd63f845e1b33b0ee7fb188167b035591f12c60a5f2db3"} Oct 30 19:30:44.033425 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:44.033274 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v" Oct 30 19:30:44.035009 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:44.034980 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg" event={"ID":"058ae453-9519-4149-80c9-8c8200d7b9f6","Type":"ContainerStarted","Data":"0673de819d032a1f6a48f2f13cdfa1f0adf96f8e115d1e19edeca1a64b8bf73c"} Oct 30 19:30:44.035198 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:44.035027 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg" event={"ID":"058ae453-9519-4149-80c9-8c8200d7b9f6","Type":"ContainerStarted","Data":"459e46c4afd87aeb82f9693b78a693e93e94e1b01772b7d75c934030c6cc676b"} Oct 30 19:30:44.035198 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:44.035083 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg" Oct 30 19:30:44.162881 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:44.162809 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v" podStartSLOduration=2.16279156 podStartE2EDuration="2.16279156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:44.160171939 +0000 UTC m=+3404.187276897" watchObservedRunningTime="2026-10-30 19:30:44.16279156 +0000 UTC m=+3404.189896586" Oct 30 19:30:44.277644 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:44.277153 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg" podStartSLOduration=2.277115509 podStartE2EDuration="2.277115509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:44.275611232 +0000 UTC m=+3404.302716193" watchObservedRunningTime="2026-10-30 19:30:44.277115509 +0000 UTC m=+3404.304220469" Oct 30 19:30:45.077485 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:45.077434 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8d65af8-g68qs"] Oct 30 19:30:45.081306 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:45.081280 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d65af8-g68qs" Oct 30 19:30:45.183027 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:45.182978 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:30:45.183027 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:45.182978 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:30:45.183027 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:45.182988 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:30:45.183418 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:45.183075 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:30:45.185906 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:45.185872 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8d65af8-g68qs"] Oct 30 19:30:45.226428 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:45.226381 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96d9b8d4-5464-4f47-984f-2af77126dede-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8d65af8-g68qs\" (UID: \"96d9b8d4-5464-4f47-984f-2af77126dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d65af8-g68qs" Oct 30 19:30:45.226428 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:45.226430 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96d9b8d4-5464-4f47-984f-2af77126dede-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8d65af8-g68qs\" (UID: \"96d9b8d4-5464-4f47-984f-2af77126dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d65af8-g68qs" Oct 30 19:30:45.226719 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:45.226542 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq978\" (UniqueName: \"kubernetes.io/projected/96d9b8d4-5464-4f47-984f-2af77126dede-kube-api-access-zq978\") pod \"maas-api-revoke-sub-b8d65af8-g68qs\" (UID: \"96d9b8d4-5464-4f47-984f-2af77126dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d65af8-g68qs" Oct 30 19:30:45.328097 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:45.328006 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96d9b8d4-5464-4f47-984f-2af77126dede-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8d65af8-g68qs\" (UID: \"96d9b8d4-5464-4f47-984f-2af77126dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d65af8-g68qs" Oct 30 19:30:45.328097 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:45.328046 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96d9b8d4-5464-4f47-984f-2af77126dede-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8d65af8-g68qs\" (UID: \"96d9b8d4-5464-4f47-984f-2af77126dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d65af8-g68qs" Oct 30 19:30:45.328097 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:45.328071 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq978\" (UniqueName: \"kubernetes.io/projected/96d9b8d4-5464-4f47-984f-2af77126dede-kube-api-access-zq978\") pod \"maas-api-revoke-sub-b8d65af8-g68qs\" (UID: \"96d9b8d4-5464-4f47-984f-2af77126dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d65af8-g68qs" Oct 30 19:30:45.328739 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:45.328712 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96d9b8d4-5464-4f47-984f-2af77126dede-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8d65af8-g68qs\" (UID: \"96d9b8d4-5464-4f47-984f-2af77126dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d65af8-g68qs" Oct 30 19:30:45.330760 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:45.330733 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96d9b8d4-5464-4f47-984f-2af77126dede-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8d65af8-g68qs\" (UID: \"96d9b8d4-5464-4f47-984f-2af77126dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d65af8-g68qs" Oct 30 19:30:45.390307 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:45.390266 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq978\" (UniqueName: \"kubernetes.io/projected/96d9b8d4-5464-4f47-984f-2af77126dede-kube-api-access-zq978\") pod \"maas-api-revoke-sub-b8d65af8-g68qs\" (UID: \"96d9b8d4-5464-4f47-984f-2af77126dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d65af8-g68qs" Oct 30 19:30:45.399290 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:45.399258 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d65af8-g68qs" Oct 30 19:30:45.776953 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:45.776922 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8d65af8-g68qs"] Oct 30 19:30:45.779307 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:30:45.779273 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d9b8d4_5464_4f47_984f_2af77126dede.slice/crio-1b5ae231602e154990b9774994e05fa38217b6f2d780d9118994278f74cb7f8d WatchSource:0}: Error finding container 1b5ae231602e154990b9774994e05fa38217b6f2d780d9118994278f74cb7f8d: Status 404 returned error can't find the container with id 1b5ae231602e154990b9774994e05fa38217b6f2d780d9118994278f74cb7f8d Oct 30 19:30:46.050346 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:46.050304 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d65af8-g68qs" event={"ID":"96d9b8d4-5464-4f47-984f-2af77126dede","Type":"ContainerStarted","Data":"1b5ae231602e154990b9774994e05fa38217b6f2d780d9118994278f74cb7f8d"} Oct 30 19:30:47.058256 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:47.058214 4531 generic.go:356] "Generic (PLEG): container finished" podID="96d9b8d4-5464-4f47-984f-2af77126dede" containerID="50926cb9eec74d69bfc26aa3057f228c30dee2e2f1408151606ec38be628fc1b" exitCode=0 Oct 30 19:30:47.058653 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:47.058266 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d65af8-g68qs" event={"ID":"96d9b8d4-5464-4f47-984f-2af77126dede","Type":"ContainerDied","Data":"50926cb9eec74d69bfc26aa3057f228c30dee2e2f1408151606ec38be628fc1b"} Oct 30 19:30:47.476086 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:47.475991 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg"] Oct 30 19:30:47.480219 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:47.480191 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg" Oct 30 19:30:47.533113 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:47.533058 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-7fdf6b-dockercfg-ftnvv\"" Oct 30 19:30:47.535258 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:47.533633 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-7fdf6b\"" Oct 30 19:30:47.545844 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:47.545734 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg"] Oct 30 19:30:47.663871 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:47.663834 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16cc1cfb-007c-4332-aad6-1d0b22eb71b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg\" (UID: \"16cc1cfb-007c-4332-aad6-1d0b22eb71b8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg" Oct 30 19:30:47.664117 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:47.663932 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhz58\" (UniqueName: \"kubernetes.io/projected/16cc1cfb-007c-4332-aad6-1d0b22eb71b8-kube-api-access-hhz58\") pod \"payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg\" (UID: \"16cc1cfb-007c-4332-aad6-1d0b22eb71b8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg" Oct 30 19:30:47.764997 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:47.764961 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhz58\" (UniqueName: \"kubernetes.io/projected/16cc1cfb-007c-4332-aad6-1d0b22eb71b8-kube-api-access-hhz58\") pod \"payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg\" (UID: \"16cc1cfb-007c-4332-aad6-1d0b22eb71b8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg" Oct 30 19:30:47.765206 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:47.765110 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16cc1cfb-007c-4332-aad6-1d0b22eb71b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg\" (UID: \"16cc1cfb-007c-4332-aad6-1d0b22eb71b8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg" Oct 30 19:30:47.765833 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:47.765804 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16cc1cfb-007c-4332-aad6-1d0b22eb71b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg\" (UID: \"16cc1cfb-007c-4332-aad6-1d0b22eb71b8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg" Oct 30 19:30:47.826391 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:47.826356 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhz58\" (UniqueName: \"kubernetes.io/projected/16cc1cfb-007c-4332-aad6-1d0b22eb71b8-kube-api-access-hhz58\") pod \"payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg\" (UID: \"16cc1cfb-007c-4332-aad6-1d0b22eb71b8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg" Oct 30 19:30:48.096464 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:48.096392 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg" Oct 30 19:30:48.236071 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:48.236040 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d65af8-g68qs" Oct 30 19:30:48.372064 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:48.371964 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96d9b8d4-5464-4f47-984f-2af77126dede-kube-api-access-zq978\" (UniqueName: \"kubernetes.io/projected/96d9b8d4-5464-4f47-984f-2af77126dede-kube-api-access-zq978\") pod \"96d9b8d4-5464-4f47-984f-2af77126dede\" (UID: \"96d9b8d4-5464-4f47-984f-2af77126dede\") " Oct 30 19:30:48.372064 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:48.372015 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96d9b8d4-5464-4f47-984f-2af77126dede-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96d9b8d4-5464-4f47-984f-2af77126dede-maas-api-xks-ca\") pod \"96d9b8d4-5464-4f47-984f-2af77126dede\" (UID: \"96d9b8d4-5464-4f47-984f-2af77126dede\") " Oct 30 19:30:48.372064 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:48.372054 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96d9b8d4-5464-4f47-984f-2af77126dede-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96d9b8d4-5464-4f47-984f-2af77126dede-maas-api-openshift-ca\") pod \"96d9b8d4-5464-4f47-984f-2af77126dede\" (UID: \"96d9b8d4-5464-4f47-984f-2af77126dede\") " Oct 30 19:30:48.372602 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:48.372572 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96d9b8d4-5464-4f47-984f-2af77126dede-maas-api-openshift-ca" pod "96d9b8d4-5464-4f47-984f-2af77126dede" (UID: "96d9b8d4-5464-4f47-984f-2af77126dede"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:48.374594 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:48.374553 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d9b8d4-5464-4f47-984f-2af77126dede-kube-api-access-zq978" pod "96d9b8d4-5464-4f47-984f-2af77126dede" (UID: "96d9b8d4-5464-4f47-984f-2af77126dede"). InnerVolumeSpecName "kube-api-access-zq978". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:48.374773 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:48.374620 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96d9b8d4-5464-4f47-984f-2af77126dede-maas-api-xks-ca" pod "96d9b8d4-5464-4f47-984f-2af77126dede" (UID: "96d9b8d4-5464-4f47-984f-2af77126dede"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:48.473722 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:48.473679 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq978\" (UniqueName: \"kubernetes.io/projected/96d9b8d4-5464-4f47-984f-2af77126dede-kube-api-access-zq978\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:48.473722 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:48.473716 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96d9b8d4-5464-4f47-984f-2af77126dede-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:48.473722 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:48.473726 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96d9b8d4-5464-4f47-984f-2af77126dede-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:30:48.494789 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:48.494758 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg"] Oct 30 19:30:48.496882 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:30:48.496857 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16cc1cfb_007c_4332_aad6_1d0b22eb71b8.slice/crio-4a00fecdd7c3fabb12407c2ac00f6f62fcb8a9670d9eea9ff74c377c0de78eb9 WatchSource:0}: Error finding container 4a00fecdd7c3fabb12407c2ac00f6f62fcb8a9670d9eea9ff74c377c0de78eb9: Status 404 returned error can't find the container with id 4a00fecdd7c3fabb12407c2ac00f6f62fcb8a9670d9eea9ff74c377c0de78eb9 Oct 30 19:30:49.072046 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:49.072014 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d65af8-g68qs" Oct 30 19:30:49.072290 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:49.072011 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8d65af8-g68qs" event={"ID":"96d9b8d4-5464-4f47-984f-2af77126dede","Type":"ContainerDied","Data":"1b5ae231602e154990b9774994e05fa38217b6f2d780d9118994278f74cb7f8d"} Oct 30 19:30:49.072290 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:49.072177 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b5ae231602e154990b9774994e05fa38217b6f2d780d9118994278f74cb7f8d" Oct 30 19:30:49.074019 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:49.073984 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg" event={"ID":"16cc1cfb-007c-4332-aad6-1d0b22eb71b8","Type":"ContainerStarted","Data":"34689f13d7c9eb15b4da3d1ca34c94092b0cad84f476690b7c7208ddea680fed"} Oct 30 19:30:49.074195 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:49.074037 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg" event={"ID":"16cc1cfb-007c-4332-aad6-1d0b22eb71b8","Type":"ContainerStarted","Data":"4a00fecdd7c3fabb12407c2ac00f6f62fcb8a9670d9eea9ff74c377c0de78eb9"} Oct 30 19:30:49.074263 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:49.074185 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg" Oct 30 19:30:49.192655 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:49.192521 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg" podStartSLOduration=2.19250543 podStartE2EDuration="2.19250543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:49.189986119 +0000 UTC m=+3409.217091078" watchObservedRunningTime="2026-10-30 19:30:49.19250543 +0000 UTC m=+3409.219610389" Oct 30 19:30:55.044965 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:55.044933 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg" Oct 30 19:30:55.045388 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:30:55.045004 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v" Oct 30 19:30:56.400215 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:30:56.400170 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:31:00.083797 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:00.083765 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg" Oct 30 19:31:07.216300 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:07.216261 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7100-b0ae9a"] Oct 30 19:31:07.216782 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:07.216735 4531 cpu_manager.go:414] "containerMap: removing container" podUID="96d9b8d4-5464-4f47-984f-2af77126dede" containerName="revoke-keys" Oct 30 19:31:07.216782 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:07.216747 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96d9b8d4-5464-4f47-984f-2af77126dede" containerName="revoke-keys" Oct 30 19:31:07.216879 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:07.216841 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="96d9b8d4-5464-4f47-984f-2af77126dede" containerName="revoke-keys" Oct 30 19:31:07.221939 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:07.221911 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7100-b0ae9a" Oct 30 19:31:07.275367 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:07.275330 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7100-b0ae9a"] Oct 30 19:31:07.288036 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:07.287996 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjblg\" (UniqueName: \"kubernetes.io/projected/a63846bb-2e11-4a43-82e7-ac381c5dbe1b-kube-api-access-vjblg\") pod \"test-curl-7100-b0ae9a\" (UID: \"a63846bb-2e11-4a43-82e7-ac381c5dbe1b\") " pod="opendatahub/test-curl-7100-b0ae9a" Oct 30 19:31:07.331347 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:07.329065 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7q8sq\"" Oct 30 19:31:07.389167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:07.389090 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjblg\" (UniqueName: \"kubernetes.io/projected/a63846bb-2e11-4a43-82e7-ac381c5dbe1b-kube-api-access-vjblg\") pod \"test-curl-7100-b0ae9a\" (UID: \"a63846bb-2e11-4a43-82e7-ac381c5dbe1b\") " pod="opendatahub/test-curl-7100-b0ae9a" Oct 30 19:31:07.450440 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:07.450397 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjblg\" (UniqueName: \"kubernetes.io/projected/a63846bb-2e11-4a43-82e7-ac381c5dbe1b-kube-api-access-vjblg\") pod \"test-curl-7100-b0ae9a\" (UID: \"a63846bb-2e11-4a43-82e7-ac381c5dbe1b\") " pod="opendatahub/test-curl-7100-b0ae9a" Oct 30 19:31:07.536800 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:07.536757 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7100-b0ae9a" Oct 30 19:31:07.915555 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:07.915528 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7100-b0ae9a"] Oct 30 19:31:07.918415 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:31:07.918377 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda63846bb_2e11_4a43_82e7_ac381c5dbe1b.slice/crio-36dd1e399a655954735afac1ba57714f4b8efa0c52c66b79043c0eb0e32ec501 WatchSource:0}: Error finding container 36dd1e399a655954735afac1ba57714f4b8efa0c52c66b79043c0eb0e32ec501: Status 404 returned error can't find the container with id 36dd1e399a655954735afac1ba57714f4b8efa0c52c66b79043c0eb0e32ec501 Oct 30 19:31:08.190017 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:08.189914 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7100-b0ae9a" event={"ID":"a63846bb-2e11-4a43-82e7-ac381c5dbe1b","Type":"ContainerStarted","Data":"c02bb8ea4edbf5a6c78e48d1748f8eb994a51b7b4062c9a33d4a6a4cedbed7b8"} Oct 30 19:31:08.190017 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:08.189964 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7100-b0ae9a" event={"ID":"a63846bb-2e11-4a43-82e7-ac381c5dbe1b","Type":"ContainerStarted","Data":"36dd1e399a655954735afac1ba57714f4b8efa0c52c66b79043c0eb0e32ec501"} Oct 30 19:31:08.307149 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:08.307069 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7100-b0ae9a" podStartSLOduration=1.30705232 podStartE2EDuration="1.30705232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:08.304357325 +0000 UTC m=+3428.331462285" watchObservedRunningTime="2026-10-30 19:31:08.30705232 +0000 UTC m=+3428.334157279" Oct 30 19:31:09.789197 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:09.789152 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7100-b0ae9a"] Oct 30 19:31:09.794299 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:09.794264 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7100-b0ae9a"] Oct 30 19:31:10.075725 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:10.075640 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cac4d782-sps87"] Oct 30 19:31:10.076106 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:10.076093 4531 cpu_manager.go:414] "containerMap: removing container" podUID="a63846bb-2e11-4a43-82e7-ac381c5dbe1b" containerName="test-curl-7100-b0ae9a" Oct 30 19:31:10.076173 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:10.076106 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a63846bb-2e11-4a43-82e7-ac381c5dbe1b" containerName="test-curl-7100-b0ae9a" Oct 30 19:31:10.076211 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:10.076206 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a63846bb-2e11-4a43-82e7-ac381c5dbe1b" containerName="test-curl-7100-b0ae9a" Oct 30 19:31:10.082159 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:10.080782 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4d782-sps87" Oct 30 19:31:10.141292 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:10.141245 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cac4d782-sps87"] Oct 30 19:31:10.187593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:10.187555 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:31:10.187593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:10.187588 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:31:10.187839 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:10.187588 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:31:10.187839 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:10.187684 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:31:10.204835 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:10.204773 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7100-b0ae9a" podUID="a63846bb-2e11-4a43-82e7-ac381c5dbe1b" containerName="test-curl-7100-b0ae9a" containerID="cri-o://c02bb8ea4edbf5a6c78e48d1748f8eb994a51b7b4062c9a33d4a6a4cedbed7b8" gracePeriod=2 Oct 30 19:31:10.223819 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:10.223771 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z46jf\" (UniqueName: \"kubernetes.io/projected/a5a092e5-a230-4564-b8f3-aeabf5580acc-kube-api-access-z46jf\") pod \"maas-api-revoke-sub-cac4d782-sps87\" (UID: \"a5a092e5-a230-4564-b8f3-aeabf5580acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4d782-sps87" Oct 30 19:31:10.224375 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:10.224332 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a092e5-a230-4564-b8f3-aeabf5580acc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cac4d782-sps87\" (UID: \"a5a092e5-a230-4564-b8f3-aeabf5580acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4d782-sps87" Oct 30 19:31:10.224503 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:10.224431 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a092e5-a230-4564-b8f3-aeabf5580acc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cac4d782-sps87\" (UID: \"a5a092e5-a230-4564-b8f3-aeabf5580acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4d782-sps87" Oct 30 19:31:10.307584 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:31:10.307542 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7100-b0ae9a\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="a63846bb-2e11-4a43-82e7-ac381c5dbe1b" pod="opendatahub/test-curl-7100-b0ae9a" Oct 30 19:31:10.325197 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:10.325160 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z46jf\" (UniqueName: \"kubernetes.io/projected/a5a092e5-a230-4564-b8f3-aeabf5580acc-kube-api-access-z46jf\") pod \"maas-api-revoke-sub-cac4d782-sps87\" (UID: \"a5a092e5-a230-4564-b8f3-aeabf5580acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4d782-sps87" Oct 30 19:31:10.325372 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:10.325238 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a092e5-a230-4564-b8f3-aeabf5580acc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cac4d782-sps87\" (UID: \"a5a092e5-a230-4564-b8f3-aeabf5580acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4d782-sps87" Oct 30 19:31:10.325372 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:10.325277 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a092e5-a230-4564-b8f3-aeabf5580acc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cac4d782-sps87\" (UID: \"a5a092e5-a230-4564-b8f3-aeabf5580acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4d782-sps87" Oct 30 19:31:10.326190 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:10.325987 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a092e5-a230-4564-b8f3-aeabf5580acc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cac4d782-sps87\" (UID: \"a5a092e5-a230-4564-b8f3-aeabf5580acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4d782-sps87" Oct 30 19:31:10.328024 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:10.327993 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a092e5-a230-4564-b8f3-aeabf5580acc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cac4d782-sps87\" (UID: \"a5a092e5-a230-4564-b8f3-aeabf5580acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4d782-sps87" Oct 30 19:31:10.387458 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:10.387414 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z46jf\" (UniqueName: \"kubernetes.io/projected/a5a092e5-a230-4564-b8f3-aeabf5580acc-kube-api-access-z46jf\") pod \"maas-api-revoke-sub-cac4d782-sps87\" (UID: \"a5a092e5-a230-4564-b8f3-aeabf5580acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4d782-sps87" Oct 30 19:31:10.396243 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:10.396212 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4d782-sps87" Oct 30 19:31:10.777723 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:10.777694 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cac4d782-sps87"] Oct 30 19:31:10.779722 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:31:10.779691 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a092e5_a230_4564_b8f3_aeabf5580acc.slice/crio-61c9abf28962c1b7a88f372751a6f6d21d6f707af2b27d3865bc8bd84a72cda7 WatchSource:0}: Error finding container 61c9abf28962c1b7a88f372751a6f6d21d6f707af2b27d3865bc8bd84a72cda7: Status 404 returned error can't find the container with id 61c9abf28962c1b7a88f372751a6f6d21d6f707af2b27d3865bc8bd84a72cda7 Oct 30 19:31:11.210398 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:11.210359 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4d782-sps87" event={"ID":"a5a092e5-a230-4564-b8f3-aeabf5580acc","Type":"ContainerStarted","Data":"3bb2c31d8bd9882153dd5af9400797f5a6f5922f81390ae66019073efd935974"} Oct 30 19:31:11.210891 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:11.210412 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4d782-sps87" event={"ID":"a5a092e5-a230-4564-b8f3-aeabf5580acc","Type":"ContainerStarted","Data":"61c9abf28962c1b7a88f372751a6f6d21d6f707af2b27d3865bc8bd84a72cda7"} Oct 30 19:31:11.330110 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:11.329995 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4d782-sps87" podStartSLOduration=1.329980911 podStartE2EDuration="1.329980911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:11.32856572 +0000 UTC m=+3431.355670705" watchObservedRunningTime="2026-10-30 19:31:11.329980911 +0000 UTC m=+3431.357085903" Oct 30 19:31:11.379483 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:31:11.379430 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7100-b0ae9a\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="a63846bb-2e11-4a43-82e7-ac381c5dbe1b" pod="opendatahub/test-curl-7100-b0ae9a" Oct 30 19:31:12.219410 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:12.219375 4531 generic.go:356] "Generic (PLEG): container finished" podID="a5a092e5-a230-4564-b8f3-aeabf5580acc" containerID="3bb2c31d8bd9882153dd5af9400797f5a6f5922f81390ae66019073efd935974" exitCode=0 Oct 30 19:31:12.219887 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:12.219451 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4d782-sps87" event={"ID":"a5a092e5-a230-4564-b8f3-aeabf5580acc","Type":"ContainerDied","Data":"3bb2c31d8bd9882153dd5af9400797f5a6f5922f81390ae66019073efd935974"} Oct 30 19:31:12.460429 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:12.460348 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7100-b0ae9a" Oct 30 19:31:12.511415 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:31:12.511366 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7100-b0ae9a\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="a63846bb-2e11-4a43-82e7-ac381c5dbe1b" pod="opendatahub/test-curl-7100-b0ae9a" Oct 30 19:31:12.549303 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:12.549252 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a63846bb-2e11-4a43-82e7-ac381c5dbe1b-kube-api-access-vjblg\" (UniqueName: \"kubernetes.io/projected/a63846bb-2e11-4a43-82e7-ac381c5dbe1b-kube-api-access-vjblg\") pod \"a63846bb-2e11-4a43-82e7-ac381c5dbe1b\" (UID: \"a63846bb-2e11-4a43-82e7-ac381c5dbe1b\") " Oct 30 19:31:12.551736 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:12.551701 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a63846bb-2e11-4a43-82e7-ac381c5dbe1b-kube-api-access-vjblg" pod "a63846bb-2e11-4a43-82e7-ac381c5dbe1b" (UID: "a63846bb-2e11-4a43-82e7-ac381c5dbe1b"). InnerVolumeSpecName "kube-api-access-vjblg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:12.650766 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:12.650708 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjblg\" (UniqueName: \"kubernetes.io/projected/a63846bb-2e11-4a43-82e7-ac381c5dbe1b-kube-api-access-vjblg\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:31:13.227104 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:13.227062 4531 generic.go:356] "Generic (PLEG): container finished" podID="a63846bb-2e11-4a43-82e7-ac381c5dbe1b" containerID="c02bb8ea4edbf5a6c78e48d1748f8eb994a51b7b4062c9a33d4a6a4cedbed7b8" exitCode=137 Oct 30 19:31:13.227104 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:13.227107 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7100-b0ae9a" Oct 30 19:31:13.227704 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:13.227173 4531 scope.go:122] "RemoveContainer" containerID="c02bb8ea4edbf5a6c78e48d1748f8eb994a51b7b4062c9a33d4a6a4cedbed7b8" Oct 30 19:31:13.238196 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:13.238168 4531 scope.go:122] "RemoveContainer" containerID="c02bb8ea4edbf5a6c78e48d1748f8eb994a51b7b4062c9a33d4a6a4cedbed7b8" Oct 30 19:31:13.238595 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:31:13.238572 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c02bb8ea4edbf5a6c78e48d1748f8eb994a51b7b4062c9a33d4a6a4cedbed7b8\": container with ID starting with c02bb8ea4edbf5a6c78e48d1748f8eb994a51b7b4062c9a33d4a6a4cedbed7b8 not found: ID does not exist" containerID="c02bb8ea4edbf5a6c78e48d1748f8eb994a51b7b4062c9a33d4a6a4cedbed7b8" Oct 30 19:31:13.238656 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:13.238605 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02bb8ea4edbf5a6c78e48d1748f8eb994a51b7b4062c9a33d4a6a4cedbed7b8"} err="failed to get container status \"c02bb8ea4edbf5a6c78e48d1748f8eb994a51b7b4062c9a33d4a6a4cedbed7b8\": rpc error: code = NotFound desc = could not find container \"c02bb8ea4edbf5a6c78e48d1748f8eb994a51b7b4062c9a33d4a6a4cedbed7b8\": container with ID starting with c02bb8ea4edbf5a6c78e48d1748f8eb994a51b7b4062c9a33d4a6a4cedbed7b8 not found: ID does not exist" Oct 30 19:31:13.278464 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:31:13.278424 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7100-b0ae9a\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="a63846bb-2e11-4a43-82e7-ac381c5dbe1b" pod="opendatahub/test-curl-7100-b0ae9a" Oct 30 19:31:13.329481 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:31:13.329425 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7100-b0ae9a\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="a63846bb-2e11-4a43-82e7-ac381c5dbe1b" pod="opendatahub/test-curl-7100-b0ae9a" Oct 30 19:31:13.392610 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:13.392568 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4d782-sps87" Oct 30 19:31:13.558969 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:31:13.558922 4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7100-b0ae9a\" is forbidden: User \"system:node:ip-10-0-133-56.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-56.us-west-1.compute.internal' and this object" podUID="a63846bb-2e11-4a43-82e7-ac381c5dbe1b" pod="opendatahub/test-curl-7100-b0ae9a" Oct 30 19:31:13.559173 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:13.559023 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a092e5-a230-4564-b8f3-aeabf5580acc-kube-api-access-z46jf\" (UniqueName: \"kubernetes.io/projected/a5a092e5-a230-4564-b8f3-aeabf5580acc-kube-api-access-z46jf\") pod \"a5a092e5-a230-4564-b8f3-aeabf5580acc\" (UID: \"a5a092e5-a230-4564-b8f3-aeabf5580acc\") " Oct 30 19:31:13.559607 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:13.559057 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5a092e5-a230-4564-b8f3-aeabf5580acc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a092e5-a230-4564-b8f3-aeabf5580acc-maas-api-xks-ca\") pod \"a5a092e5-a230-4564-b8f3-aeabf5580acc\" (UID: \"a5a092e5-a230-4564-b8f3-aeabf5580acc\") " Oct 30 19:31:13.559813 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:13.559787 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5a092e5-a230-4564-b8f3-aeabf5580acc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a092e5-a230-4564-b8f3-aeabf5580acc-maas-api-openshift-ca\") pod \"a5a092e5-a230-4564-b8f3-aeabf5580acc\" (UID: \"a5a092e5-a230-4564-b8f3-aeabf5580acc\") " Oct 30 19:31:13.560148 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:13.560098 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5a092e5-a230-4564-b8f3-aeabf5580acc-maas-api-openshift-ca" pod "a5a092e5-a230-4564-b8f3-aeabf5580acc" (UID: "a5a092e5-a230-4564-b8f3-aeabf5580acc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:13.561272 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:13.561242 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5a092e5-a230-4564-b8f3-aeabf5580acc-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:31:13.562239 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:13.562208 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5a092e5-a230-4564-b8f3-aeabf5580acc-maas-api-xks-ca" pod "a5a092e5-a230-4564-b8f3-aeabf5580acc" (UID: "a5a092e5-a230-4564-b8f3-aeabf5580acc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:31:13.562415 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:13.562385 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a092e5-a230-4564-b8f3-aeabf5580acc-kube-api-access-z46jf" pod "a5a092e5-a230-4564-b8f3-aeabf5580acc" (UID: "a5a092e5-a230-4564-b8f3-aeabf5580acc"). InnerVolumeSpecName "kube-api-access-z46jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:13.663325 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:13.663284 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5a092e5-a230-4564-b8f3-aeabf5580acc-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:31:13.663325 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:13.663324 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z46jf\" (UniqueName: \"kubernetes.io/projected/a5a092e5-a230-4564-b8f3-aeabf5580acc-kube-api-access-z46jf\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:31:14.098367 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:14.098330 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a63846bb-2e11-4a43-82e7-ac381c5dbe1b" path="/var/lib/kubelet/pods/a63846bb-2e11-4a43-82e7-ac381c5dbe1b/volumes" Oct 30 19:31:14.234445 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:14.234414 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4d782-sps87" Oct 30 19:31:14.234959 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:14.234440 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4d782-sps87" event={"ID":"a5a092e5-a230-4564-b8f3-aeabf5580acc","Type":"ContainerDied","Data":"61c9abf28962c1b7a88f372751a6f6d21d6f707af2b27d3865bc8bd84a72cda7"} Oct 30 19:31:14.234959 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:14.234504 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61c9abf28962c1b7a88f372751a6f6d21d6f707af2b27d3865bc8bd84a72cda7" Oct 30 19:31:15.208709 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:15.208665 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-200c46-gb76s"] Oct 30 19:31:15.209192 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:15.209176 4531 cpu_manager.go:414] "containerMap: removing container" podUID="a5a092e5-a230-4564-b8f3-aeabf5580acc" containerName="revoke-keys" Oct 30 19:31:15.209250 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:15.209192 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a092e5-a230-4564-b8f3-aeabf5580acc" containerName="revoke-keys" Oct 30 19:31:15.209302 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:15.209291 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a092e5-a230-4564-b8f3-aeabf5580acc" containerName="revoke-keys" Oct 30 19:31:15.212728 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:15.212706 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-200c46-gb76s" Oct 30 19:31:15.266093 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:15.266060 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:31:15.266564 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:15.266118 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:31:15.266564 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:15.266118 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:31:15.296332 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:15.296299 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:31:15.318428 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:15.318385 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-200c46-gb76s"] Oct 30 19:31:15.384806 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:15.384766 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gt27\" (UniqueName: \"kubernetes.io/projected/bbed56a4-dd80-4d21-9121-f2c99b8f5dc7-kube-api-access-6gt27\") pod \"maas-api-revoke-keys-e2e-explicit-200c46-gb76s\" (UID: \"bbed56a4-dd80-4d21-9121-f2c99b8f5dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-200c46-gb76s" Oct 30 19:31:15.384986 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:15.384817 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbed56a4-dd80-4d21-9121-f2c99b8f5dc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-200c46-gb76s\" (UID: \"bbed56a4-dd80-4d21-9121-f2c99b8f5dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-200c46-gb76s" Oct 30 19:31:15.385049 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:15.384973 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbed56a4-dd80-4d21-9121-f2c99b8f5dc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-200c46-gb76s\" (UID: \"bbed56a4-dd80-4d21-9121-f2c99b8f5dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-200c46-gb76s" Oct 30 19:31:15.486167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:15.486006 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gt27\" (UniqueName: \"kubernetes.io/projected/bbed56a4-dd80-4d21-9121-f2c99b8f5dc7-kube-api-access-6gt27\") pod \"maas-api-revoke-keys-e2e-explicit-200c46-gb76s\" (UID: \"bbed56a4-dd80-4d21-9121-f2c99b8f5dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-200c46-gb76s" Oct 30 19:31:15.486167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:15.486079 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbed56a4-dd80-4d21-9121-f2c99b8f5dc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-200c46-gb76s\" (UID: \"bbed56a4-dd80-4d21-9121-f2c99b8f5dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-200c46-gb76s" Oct 30 19:31:15.486436 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:15.486232 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbed56a4-dd80-4d21-9121-f2c99b8f5dc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-200c46-gb76s\" (UID: \"bbed56a4-dd80-4d21-9121-f2c99b8f5dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-200c46-gb76s" Oct 30 19:31:15.486861 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:15.486840 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbed56a4-dd80-4d21-9121-f2c99b8f5dc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-200c46-gb76s\" (UID: \"bbed56a4-dd80-4d21-9121-f2c99b8f5dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-200c46-gb76s" Oct 30 19:31:15.488800 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:15.488775 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbed56a4-dd80-4d21-9121-f2c99b8f5dc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-200c46-gb76s\" (UID: \"bbed56a4-dd80-4d21-9121-f2c99b8f5dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-200c46-gb76s" Oct 30 19:31:15.547912 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:15.547872 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gt27\" (UniqueName: \"kubernetes.io/projected/bbed56a4-dd80-4d21-9121-f2c99b8f5dc7-kube-api-access-6gt27\") pod \"maas-api-revoke-keys-e2e-explicit-200c46-gb76s\" (UID: \"bbed56a4-dd80-4d21-9121-f2c99b8f5dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-200c46-gb76s" Oct 30 19:31:15.827439 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:15.827393 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-200c46-gb76s" Oct 30 19:31:16.205781 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:16.205673 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-200c46-gb76s"] Oct 30 19:31:16.208346 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:31:16.208315 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbed56a4_dd80_4d21_9121_f2c99b8f5dc7.slice/crio-9e78096a3799810b66e74d97a86e59bf80ed24256c1cb9cfd5daa566bc5419ef WatchSource:0}: Error finding container 9e78096a3799810b66e74d97a86e59bf80ed24256c1cb9cfd5daa566bc5419ef: Status 404 returned error can't find the container with id 9e78096a3799810b66e74d97a86e59bf80ed24256c1cb9cfd5daa566bc5419ef Oct 30 19:31:16.248977 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:16.248942 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-200c46-gb76s" event={"ID":"bbed56a4-dd80-4d21-9121-f2c99b8f5dc7","Type":"ContainerStarted","Data":"9e78096a3799810b66e74d97a86e59bf80ed24256c1cb9cfd5daa566bc5419ef"} Oct 30 19:31:17.256874 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:17.256834 4531 generic.go:356] "Generic (PLEG): container finished" podID="bbed56a4-dd80-4d21-9121-f2c99b8f5dc7" containerID="9acb82373335e605c8b2c1c63ffd179d692737e24b28aff5ad77ffa0a85bb634" exitCode=0 Oct 30 19:31:17.257309 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:17.256894 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-200c46-gb76s" event={"ID":"bbed56a4-dd80-4d21-9121-f2c99b8f5dc7","Type":"ContainerDied","Data":"9acb82373335e605c8b2c1c63ffd179d692737e24b28aff5ad77ffa0a85bb634"} Oct 30 19:31:18.419483 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:18.419451 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-200c46-gb76s" Oct 30 19:31:18.519767 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:18.519661 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbed56a4-dd80-4d21-9121-f2c99b8f5dc7-kube-api-access-6gt27\" (UniqueName: \"kubernetes.io/projected/bbed56a4-dd80-4d21-9121-f2c99b8f5dc7-kube-api-access-6gt27\") pod \"bbed56a4-dd80-4d21-9121-f2c99b8f5dc7\" (UID: \"bbed56a4-dd80-4d21-9121-f2c99b8f5dc7\") " Oct 30 19:31:18.519954 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:18.519782 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbed56a4-dd80-4d21-9121-f2c99b8f5dc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbed56a4-dd80-4d21-9121-f2c99b8f5dc7-maas-api-openshift-ca\") pod \"bbed56a4-dd80-4d21-9121-f2c99b8f5dc7\" (UID: \"bbed56a4-dd80-4d21-9121-f2c99b8f5dc7\") " Oct 30 19:31:18.519954 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:18.519817 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbed56a4-dd80-4d21-9121-f2c99b8f5dc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbed56a4-dd80-4d21-9121-f2c99b8f5dc7-maas-api-xks-ca\") pod \"bbed56a4-dd80-4d21-9121-f2c99b8f5dc7\" (UID: \"bbed56a4-dd80-4d21-9121-f2c99b8f5dc7\") " Oct 30 19:31:18.520253 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:18.520229 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbed56a4-dd80-4d21-9121-f2c99b8f5dc7-maas-api-openshift-ca" pod "bbed56a4-dd80-4d21-9121-f2c99b8f5dc7" (UID: "bbed56a4-dd80-4d21-9121-f2c99b8f5dc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:18.522339 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:18.522309 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbed56a4-dd80-4d21-9121-f2c99b8f5dc7-maas-api-xks-ca" pod "bbed56a4-dd80-4d21-9121-f2c99b8f5dc7" (UID: "bbed56a4-dd80-4d21-9121-f2c99b8f5dc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:31:18.522339 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:18.522325 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbed56a4-dd80-4d21-9121-f2c99b8f5dc7-kube-api-access-6gt27" pod "bbed56a4-dd80-4d21-9121-f2c99b8f5dc7" (UID: "bbed56a4-dd80-4d21-9121-f2c99b8f5dc7"). InnerVolumeSpecName "kube-api-access-6gt27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:18.620901 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:18.620855 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gt27\" (UniqueName: \"kubernetes.io/projected/bbed56a4-dd80-4d21-9121-f2c99b8f5dc7-kube-api-access-6gt27\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:31:18.620901 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:18.620893 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbed56a4-dd80-4d21-9121-f2c99b8f5dc7-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:31:18.620901 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:18.620904 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbed56a4-dd80-4d21-9121-f2c99b8f5dc7-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:31:19.280474 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:19.280427 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-200c46-gb76s" event={"ID":"bbed56a4-dd80-4d21-9121-f2c99b8f5dc7","Type":"ContainerDied","Data":"9e78096a3799810b66e74d97a86e59bf80ed24256c1cb9cfd5daa566bc5419ef"} Oct 30 19:31:19.280681 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:19.280531 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e78096a3799810b66e74d97a86e59bf80ed24256c1cb9cfd5daa566bc5419ef" Oct 30 19:31:19.280681 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:19.280538 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-200c46-gb76s" Oct 30 19:31:20.909622 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:20.909579 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7791f32-bghtn"] Oct 30 19:31:20.910167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:20.910147 4531 cpu_manager.go:414] "containerMap: removing container" podUID="bbed56a4-dd80-4d21-9121-f2c99b8f5dc7" containerName="revoke-keys" Oct 30 19:31:20.910257 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:20.910167 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbed56a4-dd80-4d21-9121-f2c99b8f5dc7" containerName="revoke-keys" Oct 30 19:31:20.910307 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:20.910277 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbed56a4-dd80-4d21-9121-f2c99b8f5dc7" containerName="revoke-keys" Oct 30 19:31:20.913617 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:20.913594 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7791f32-bghtn" Oct 30 19:31:20.939550 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:20.939509 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e081c9c-cfd8-41bb-9aab-60e216e9ef1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7791f32-bghtn\" (UID: \"1e081c9c-cfd8-41bb-9aab-60e216e9ef1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7791f32-bghtn" Oct 30 19:31:20.939739 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:20.939594 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e081c9c-cfd8-41bb-9aab-60e216e9ef1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7791f32-bghtn\" (UID: \"1e081c9c-cfd8-41bb-9aab-60e216e9ef1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7791f32-bghtn" Oct 30 19:31:20.939739 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:20.939661 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx7ds\" (UniqueName: \"kubernetes.io/projected/1e081c9c-cfd8-41bb-9aab-60e216e9ef1a-kube-api-access-bx7ds\") pod \"maas-api-revoke-sub-c7791f32-bghtn\" (UID: \"1e081c9c-cfd8-41bb-9aab-60e216e9ef1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7791f32-bghtn" Oct 30 19:31:21.022419 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:21.022378 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:31:21.022419 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:21.022378 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:31:21.022419 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:21.022378 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:31:21.022419 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:21.022433 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:31:21.025337 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:21.025304 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7791f32-bghtn"] Oct 30 19:31:21.041020 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:21.040974 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e081c9c-cfd8-41bb-9aab-60e216e9ef1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7791f32-bghtn\" (UID: \"1e081c9c-cfd8-41bb-9aab-60e216e9ef1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7791f32-bghtn" Oct 30 19:31:21.041228 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:21.041034 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx7ds\" (UniqueName: \"kubernetes.io/projected/1e081c9c-cfd8-41bb-9aab-60e216e9ef1a-kube-api-access-bx7ds\") pod \"maas-api-revoke-sub-c7791f32-bghtn\" (UID: \"1e081c9c-cfd8-41bb-9aab-60e216e9ef1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7791f32-bghtn" Oct 30 19:31:21.041228 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:21.041117 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e081c9c-cfd8-41bb-9aab-60e216e9ef1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7791f32-bghtn\" (UID: \"1e081c9c-cfd8-41bb-9aab-60e216e9ef1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7791f32-bghtn" Oct 30 19:31:21.041790 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:21.041770 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e081c9c-cfd8-41bb-9aab-60e216e9ef1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7791f32-bghtn\" (UID: \"1e081c9c-cfd8-41bb-9aab-60e216e9ef1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7791f32-bghtn" Oct 30 19:31:21.043725 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:21.043703 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e081c9c-cfd8-41bb-9aab-60e216e9ef1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7791f32-bghtn\" (UID: \"1e081c9c-cfd8-41bb-9aab-60e216e9ef1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7791f32-bghtn" Oct 30 19:31:21.104206 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:21.104163 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx7ds\" (UniqueName: \"kubernetes.io/projected/1e081c9c-cfd8-41bb-9aab-60e216e9ef1a-kube-api-access-bx7ds\") pod \"maas-api-revoke-sub-c7791f32-bghtn\" (UID: \"1e081c9c-cfd8-41bb-9aab-60e216e9ef1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7791f32-bghtn" Oct 30 19:31:21.228570 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:21.228473 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7791f32-bghtn" Oct 30 19:31:21.615722 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:21.615689 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7791f32-bghtn"] Oct 30 19:31:21.617657 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:31:21.617606 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e081c9c_cfd8_41bb_9aab_60e216e9ef1a.slice/crio-032107e614336478cb6e66c8e40aaeb0b6133015fb134fc1f72d3366b2ccf05a WatchSource:0}: Error finding container 032107e614336478cb6e66c8e40aaeb0b6133015fb134fc1f72d3366b2ccf05a: Status 404 returned error can't find the container with id 032107e614336478cb6e66c8e40aaeb0b6133015fb134fc1f72d3366b2ccf05a Oct 30 19:31:22.302796 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:22.302757 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7791f32-bghtn" event={"ID":"1e081c9c-cfd8-41bb-9aab-60e216e9ef1a","Type":"ContainerStarted","Data":"98e0883896194694e2a4b264169f304a417378afeea8b0d10dc1d2e6dd61efa9"} Oct 30 19:31:22.303245 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:22.302815 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7791f32-bghtn" event={"ID":"1e081c9c-cfd8-41bb-9aab-60e216e9ef1a","Type":"ContainerStarted","Data":"032107e614336478cb6e66c8e40aaeb0b6133015fb134fc1f72d3366b2ccf05a"} Oct 30 19:31:22.424562 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:22.424502 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7791f32-bghtn" podStartSLOduration=2.424480476 podStartE2EDuration="2.424480476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:22.423519679 +0000 UTC m=+3442.450624640" watchObservedRunningTime="2026-10-30 19:31:22.424480476 +0000 UTC m=+3442.451585439" Oct 30 19:31:23.311418 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:23.311377 4531 generic.go:356] "Generic (PLEG): container finished" podID="1e081c9c-cfd8-41bb-9aab-60e216e9ef1a" containerID="98e0883896194694e2a4b264169f304a417378afeea8b0d10dc1d2e6dd61efa9" exitCode=0 Oct 30 19:31:23.311844 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:23.311465 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7791f32-bghtn" event={"ID":"1e081c9c-cfd8-41bb-9aab-60e216e9ef1a","Type":"ContainerDied","Data":"98e0883896194694e2a4b264169f304a417378afeea8b0d10dc1d2e6dd61efa9"} Oct 30 19:31:24.475707 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:24.475672 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7791f32-bghtn" Oct 30 19:31:24.582789 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:24.582698 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e081c9c-cfd8-41bb-9aab-60e216e9ef1a-kube-api-access-bx7ds\" (UniqueName: \"kubernetes.io/projected/1e081c9c-cfd8-41bb-9aab-60e216e9ef1a-kube-api-access-bx7ds\") pod \"1e081c9c-cfd8-41bb-9aab-60e216e9ef1a\" (UID: \"1e081c9c-cfd8-41bb-9aab-60e216e9ef1a\") " Oct 30 19:31:24.582964 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:24.582851 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e081c9c-cfd8-41bb-9aab-60e216e9ef1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e081c9c-cfd8-41bb-9aab-60e216e9ef1a-maas-api-xks-ca\") pod \"1e081c9c-cfd8-41bb-9aab-60e216e9ef1a\" (UID: \"1e081c9c-cfd8-41bb-9aab-60e216e9ef1a\") " Oct 30 19:31:24.582964 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:24.582901 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e081c9c-cfd8-41bb-9aab-60e216e9ef1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e081c9c-cfd8-41bb-9aab-60e216e9ef1a-maas-api-openshift-ca\") pod \"1e081c9c-cfd8-41bb-9aab-60e216e9ef1a\" (UID: \"1e081c9c-cfd8-41bb-9aab-60e216e9ef1a\") " Oct 30 19:31:24.583286 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:24.583259 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e081c9c-cfd8-41bb-9aab-60e216e9ef1a-maas-api-openshift-ca" pod "1e081c9c-cfd8-41bb-9aab-60e216e9ef1a" (UID: "1e081c9c-cfd8-41bb-9aab-60e216e9ef1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:24.585320 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:24.585291 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e081c9c-cfd8-41bb-9aab-60e216e9ef1a-maas-api-xks-ca" pod "1e081c9c-cfd8-41bb-9aab-60e216e9ef1a" (UID: "1e081c9c-cfd8-41bb-9aab-60e216e9ef1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:31:24.585444 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:24.585391 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e081c9c-cfd8-41bb-9aab-60e216e9ef1a-kube-api-access-bx7ds" pod "1e081c9c-cfd8-41bb-9aab-60e216e9ef1a" (UID: "1e081c9c-cfd8-41bb-9aab-60e216e9ef1a"). InnerVolumeSpecName "kube-api-access-bx7ds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:24.684452 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:24.684411 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e081c9c-cfd8-41bb-9aab-60e216e9ef1a-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:31:24.684452 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:24.684450 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e081c9c-cfd8-41bb-9aab-60e216e9ef1a-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:31:24.684452 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:24.684462 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx7ds\" (UniqueName: \"kubernetes.io/projected/1e081c9c-cfd8-41bb-9aab-60e216e9ef1a-kube-api-access-bx7ds\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:31:25.326774 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:25.326738 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7791f32-bghtn" Oct 30 19:31:25.326975 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:25.326774 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7791f32-bghtn" event={"ID":"1e081c9c-cfd8-41bb-9aab-60e216e9ef1a","Type":"ContainerDied","Data":"032107e614336478cb6e66c8e40aaeb0b6133015fb134fc1f72d3366b2ccf05a"} Oct 30 19:31:25.326975 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:25.326823 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="032107e614336478cb6e66c8e40aaeb0b6133015fb134fc1f72d3366b2ccf05a" Oct 30 19:31:26.399488 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:31:26.399449 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:31:47.664207 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:47.664109 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj"] Oct 30 19:31:47.664670 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:47.664610 4531 cpu_manager.go:414] "containerMap: removing container" podUID="1e081c9c-cfd8-41bb-9aab-60e216e9ef1a" containerName="revoke-keys" Oct 30 19:31:47.664670 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:47.664623 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e081c9c-cfd8-41bb-9aab-60e216e9ef1a" containerName="revoke-keys" Oct 30 19:31:47.664747 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:47.664703 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e081c9c-cfd8-41bb-9aab-60e216e9ef1a" containerName="revoke-keys" Oct 30 19:31:47.668054 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:47.668036 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj" Oct 30 19:31:47.720480 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:47.720452 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-cdd844\"" Oct 30 19:31:47.757285 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:47.757228 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0348ded6-6eb6-4d7b-80b1-aa551aae81eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj\" (UID: \"0348ded6-6eb6-4d7b-80b1-aa551aae81eb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj" Oct 30 19:31:47.757538 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:47.757329 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98bhn\" (UniqueName: \"kubernetes.io/projected/0348ded6-6eb6-4d7b-80b1-aa551aae81eb-kube-api-access-98bhn\") pod \"payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj\" (UID: \"0348ded6-6eb6-4d7b-80b1-aa551aae81eb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj" Oct 30 19:31:47.771938 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:47.771901 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-cdd844-dockercfg-kz79t\"" Oct 30 19:31:47.774169 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:47.774118 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj"] Oct 30 19:31:47.796263 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:47.796222 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7"] Oct 30 19:31:47.801100 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:47.801060 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7" Oct 30 19:31:47.850936 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:47.850894 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7"] Oct 30 19:31:47.858651 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:47.858604 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1929b710-1a08-4c73-b6fc-ecceac5089f5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7\" (UID: \"1929b710-1a08-4c73-b6fc-ecceac5089f5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7" Oct 30 19:31:47.858865 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:47.858740 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwmht\" (UniqueName: \"kubernetes.io/projected/1929b710-1a08-4c73-b6fc-ecceac5089f5-kube-api-access-rwmht\") pod \"payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7\" (UID: \"1929b710-1a08-4c73-b6fc-ecceac5089f5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7" Oct 30 19:31:47.858961 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:47.858902 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0348ded6-6eb6-4d7b-80b1-aa551aae81eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj\" (UID: \"0348ded6-6eb6-4d7b-80b1-aa551aae81eb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj" Oct 30 19:31:47.859024 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:47.858982 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98bhn\" (UniqueName: \"kubernetes.io/projected/0348ded6-6eb6-4d7b-80b1-aa551aae81eb-kube-api-access-98bhn\") pod \"payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj\" (UID: \"0348ded6-6eb6-4d7b-80b1-aa551aae81eb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj" Oct 30 19:31:47.860331 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:47.860304 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0348ded6-6eb6-4d7b-80b1-aa551aae81eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj\" (UID: \"0348ded6-6eb6-4d7b-80b1-aa551aae81eb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj" Oct 30 19:31:47.920725 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:47.920637 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98bhn\" (UniqueName: \"kubernetes.io/projected/0348ded6-6eb6-4d7b-80b1-aa551aae81eb-kube-api-access-98bhn\") pod \"payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj\" (UID: \"0348ded6-6eb6-4d7b-80b1-aa551aae81eb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj" Oct 30 19:31:47.959897 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:47.959846 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1929b710-1a08-4c73-b6fc-ecceac5089f5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7\" (UID: \"1929b710-1a08-4c73-b6fc-ecceac5089f5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7" Oct 30 19:31:47.960109 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:47.959905 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwmht\" (UniqueName: \"kubernetes.io/projected/1929b710-1a08-4c73-b6fc-ecceac5089f5-kube-api-access-rwmht\") pod \"payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7\" (UID: \"1929b710-1a08-4c73-b6fc-ecceac5089f5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7" Oct 30 19:31:47.960547 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:47.960526 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1929b710-1a08-4c73-b6fc-ecceac5089f5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7\" (UID: \"1929b710-1a08-4c73-b6fc-ecceac5089f5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7" Oct 30 19:31:47.983383 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:47.983352 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj" Oct 30 19:31:48.021638 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:48.021599 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwmht\" (UniqueName: \"kubernetes.io/projected/1929b710-1a08-4c73-b6fc-ecceac5089f5-kube-api-access-rwmht\") pod \"payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7\" (UID: \"1929b710-1a08-4c73-b6fc-ecceac5089f5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7" Oct 30 19:31:48.117484 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:48.117437 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7" Oct 30 19:31:48.365159 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:48.365096 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj"] Oct 30 19:31:48.366682 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:31:48.366658 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0348ded6_6eb6_4d7b_80b1_aa551aae81eb.slice/crio-88946e2fcaa93b6c6c8361724b3e78dea4daf2e8628098a34a486d409ed0728a WatchSource:0}: Error finding container 88946e2fcaa93b6c6c8361724b3e78dea4daf2e8628098a34a486d409ed0728a: Status 404 returned error can't find the container with id 88946e2fcaa93b6c6c8361724b3e78dea4daf2e8628098a34a486d409ed0728a Oct 30 19:31:48.469731 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:48.469694 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj" event={"ID":"0348ded6-6eb6-4d7b-80b1-aa551aae81eb","Type":"ContainerStarted","Data":"a95d4422bb10ed45d9b450f5aaffdb70a128e78cd8ac0996ce437c3cbc862bba"} Oct 30 19:31:48.469932 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:48.469760 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj" event={"ID":"0348ded6-6eb6-4d7b-80b1-aa551aae81eb","Type":"ContainerStarted","Data":"88946e2fcaa93b6c6c8361724b3e78dea4daf2e8628098a34a486d409ed0728a"} Oct 30 19:31:48.469932 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:48.469788 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj" Oct 30 19:31:48.499539 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:48.499510 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7"] Oct 30 19:31:48.501389 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:31:48.501355 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1929b710_1a08_4c73_b6fc_ecceac5089f5.slice/crio-537b1035892ffddc863fe5bd6398ff7a6ff6f1e26cfa041a273d1aaf7d0b9010 WatchSource:0}: Error finding container 537b1035892ffddc863fe5bd6398ff7a6ff6f1e26cfa041a273d1aaf7d0b9010: Status 404 returned error can't find the container with id 537b1035892ffddc863fe5bd6398ff7a6ff6f1e26cfa041a273d1aaf7d0b9010 Oct 30 19:31:48.586778 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:48.586711 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj" podStartSLOduration=1.586658052 podStartE2EDuration="1.586658052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:48.585349591 +0000 UTC m=+3468.612454565" watchObservedRunningTime="2026-10-30 19:31:48.586658052 +0000 UTC m=+3468.613763011" Oct 30 19:31:49.477398 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:49.477360 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7" event={"ID":"1929b710-1a08-4c73-b6fc-ecceac5089f5","Type":"ContainerStarted","Data":"c28bacaf4def14b9fce10f834ab29ab08f7fcbcc32336583559ef4112b37690a"} Oct 30 19:31:49.477926 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:49.477431 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7" event={"ID":"1929b710-1a08-4c73-b6fc-ecceac5089f5","Type":"ContainerStarted","Data":"537b1035892ffddc863fe5bd6398ff7a6ff6f1e26cfa041a273d1aaf7d0b9010"} Oct 30 19:31:49.477926 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:49.477591 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7" Oct 30 19:31:56.399471 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:31:56.399430 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:31:59.481071 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:59.481034 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj" Oct 30 19:31:59.597061 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:31:59.597007 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7" podStartSLOduration=12.596991087 podStartE2EDuration="12.596991087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:49.596227268 +0000 UTC m=+3469.623332227" watchObservedRunningTime="2026-10-30 19:31:59.596991087 +0000 UTC m=+3479.624096057" Oct 30 19:32:00.487271 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:32:00.487237 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7" Oct 30 19:32:22.920874 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:32:22.920834 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-200c46-gb76s"] Oct 30 19:32:22.946298 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:32:22.946258 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-200c46-gb76s"] Oct 30 19:32:24.098330 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:32:24.098282 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbed56a4-dd80-4d21-9121-f2c99b8f5dc7" path="/var/lib/kubelet/pods/bbed56a4-dd80-4d21-9121-f2c99b8f5dc7/volumes" Oct 30 19:32:26.400559 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:32:26.400512 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:32:46.051426 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:32:46.051380 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc43760f-sdcj7"] Oct 30 19:32:46.101331 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:32:46.101297 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc43760f-sdcj7"] Oct 30 19:32:48.099316 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:32:48.099273 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf335c92-b015-4e59-9142-181bd2e7aee1" path="/var/lib/kubelet/pods/cf335c92-b015-4e59-9142-181bd2e7aee1/volumes" Oct 30 19:32:56.397853 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:32:56.397813 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:33:02.929027 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:02.928993 4531 scope.go:122] "RemoveContainer" containerID="d26bd5cf91f3d730e5adb28d86464ba053a8e25b3531f6a8c97908de237714e3" Oct 30 19:33:14.670361 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:14.670232 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j"] Oct 30 19:33:14.675214 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:14.675179 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j" Oct 30 19:33:14.723824 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:14.723778 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j"] Oct 30 19:33:14.747230 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:14.747186 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5gqn\" (UniqueName: \"kubernetes.io/projected/c982a4ff-8174-4543-930b-20290e9dcb48-kube-api-access-n5gqn\") pod \"payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j\" (UID: \"c982a4ff-8174-4543-930b-20290e9dcb48\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j" Oct 30 19:33:14.747412 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:14.747339 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c982a4ff-8174-4543-930b-20290e9dcb48-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j\" (UID: \"c982a4ff-8174-4543-930b-20290e9dcb48\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j" Oct 30 19:33:14.768771 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:14.768736 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-0f83f5\"" Oct 30 19:33:14.768953 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:14.768780 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-0f83f5-dockercfg-n56rf\"" Oct 30 19:33:14.849073 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:14.849031 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c982a4ff-8174-4543-930b-20290e9dcb48-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j\" (UID: \"c982a4ff-8174-4543-930b-20290e9dcb48\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j" Oct 30 19:33:14.849291 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:14.849269 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5gqn\" (UniqueName: \"kubernetes.io/projected/c982a4ff-8174-4543-930b-20290e9dcb48-kube-api-access-n5gqn\") pod \"payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j\" (UID: \"c982a4ff-8174-4543-930b-20290e9dcb48\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j" Oct 30 19:33:14.849904 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:14.849882 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c982a4ff-8174-4543-930b-20290e9dcb48-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j\" (UID: \"c982a4ff-8174-4543-930b-20290e9dcb48\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j" Oct 30 19:33:14.910384 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:14.910348 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5gqn\" (UniqueName: \"kubernetes.io/projected/c982a4ff-8174-4543-930b-20290e9dcb48-kube-api-access-n5gqn\") pod \"payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j\" (UID: \"c982a4ff-8174-4543-930b-20290e9dcb48\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j" Oct 30 19:33:15.010864 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:15.010831 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j" Oct 30 19:33:15.387549 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:15.387515 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j"] Oct 30 19:33:15.391937 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:33:15.391900 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc982a4ff_8174_4543_930b_20290e9dcb48.slice/crio-390277e1b237e7af32275c2c9dee71f69d7ac846d194ac80f79292126c6d84bc WatchSource:0}: Error finding container 390277e1b237e7af32275c2c9dee71f69d7ac846d194ac80f79292126c6d84bc: Status 404 returned error can't find the container with id 390277e1b237e7af32275c2c9dee71f69d7ac846d194ac80f79292126c6d84bc Oct 30 19:33:16.035680 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:16.035635 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j" event={"ID":"c982a4ff-8174-4543-930b-20290e9dcb48","Type":"ContainerStarted","Data":"74cd303636f1cafe17d04bb0b9923357d5bd4237f5cec3a442bf772737ef1cdc"} Oct 30 19:33:16.036120 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:16.035706 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j" event={"ID":"c982a4ff-8174-4543-930b-20290e9dcb48","Type":"ContainerStarted","Data":"390277e1b237e7af32275c2c9dee71f69d7ac846d194ac80f79292126c6d84bc"} Oct 30 19:33:16.036120 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:16.035729 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j" Oct 30 19:33:16.156620 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:16.156562 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j" podStartSLOduration=2.156544978 podStartE2EDuration="2.156544978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:16.153435273 +0000 UTC m=+3556.180540232" watchObservedRunningTime="2026-10-30 19:33:16.156544978 +0000 UTC m=+3556.183649936" Oct 30 19:33:26.397667 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:33:26.397635 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:33:27.045683 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:27.045654 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j" Oct 30 19:33:27.608047 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:27.608008 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k"] Oct 30 19:33:27.612593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:27.612541 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k" Oct 30 19:33:27.674038 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:27.674007 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k"] Oct 30 19:33:27.719311 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:27.719268 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-185e8b\"" Oct 30 19:33:27.719311 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:27.719312 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-185e8b-dockercfg-6xj6c\"" Oct 30 19:33:27.802059 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:27.802011 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lsbs\" (UniqueName: \"kubernetes.io/projected/99261072-41ec-47ab-976f-969826789970-kube-api-access-2lsbs\") pod \"payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k\" (UID: \"99261072-41ec-47ab-976f-969826789970\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k" Oct 30 19:33:27.802059 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:27.802058 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99261072-41ec-47ab-976f-969826789970-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k\" (UID: \"99261072-41ec-47ab-976f-969826789970\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k" Oct 30 19:33:27.903319 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:27.903226 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lsbs\" (UniqueName: \"kubernetes.io/projected/99261072-41ec-47ab-976f-969826789970-kube-api-access-2lsbs\") pod \"payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k\" (UID: \"99261072-41ec-47ab-976f-969826789970\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k" Oct 30 19:33:27.903319 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:27.903282 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99261072-41ec-47ab-976f-969826789970-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k\" (UID: \"99261072-41ec-47ab-976f-969826789970\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k" Oct 30 19:33:27.904052 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:27.904031 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99261072-41ec-47ab-976f-969826789970-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k\" (UID: \"99261072-41ec-47ab-976f-969826789970\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k" Oct 30 19:33:27.965010 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:27.964972 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lsbs\" (UniqueName: \"kubernetes.io/projected/99261072-41ec-47ab-976f-969826789970-kube-api-access-2lsbs\") pod \"payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k\" (UID: \"99261072-41ec-47ab-976f-969826789970\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k" Oct 30 19:33:28.244576 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:28.244539 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k" Oct 30 19:33:28.623752 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:28.623719 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k"] Oct 30 19:33:28.626167 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:33:28.626123 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99261072_41ec_47ab_976f_969826789970.slice/crio-3e780aff814466982f7e10a3770b756574835464135bb0e248b138c61223a74e WatchSource:0}: Error finding container 3e780aff814466982f7e10a3770b756574835464135bb0e248b138c61223a74e: Status 404 returned error can't find the container with id 3e780aff814466982f7e10a3770b756574835464135bb0e248b138c61223a74e Oct 30 19:33:29.116891 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:29.116847 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k" event={"ID":"99261072-41ec-47ab-976f-969826789970","Type":"ContainerStarted","Data":"8532d3300dd437d76cbf20f3eca2ba814f49dcb640f1f548281ae4839312744e"} Oct 30 19:33:29.117095 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:29.116920 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k" Oct 30 19:33:29.117095 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:29.116942 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k" event={"ID":"99261072-41ec-47ab-976f-969826789970","Type":"ContainerStarted","Data":"3e780aff814466982f7e10a3770b756574835464135bb0e248b138c61223a74e"} Oct 30 19:33:29.236737 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:29.236674 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k" podStartSLOduration=2.236655718 podStartE2EDuration="2.236655718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:29.233825799 +0000 UTC m=+3569.260930761" watchObservedRunningTime="2026-10-30 19:33:29.236655718 +0000 UTC m=+3569.263760678" Oct 30 19:33:34.126662 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.126624 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw"] Oct 30 19:33:34.130251 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.130228 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" Oct 30 19:33:34.166180 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.166105 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n25g\" (UniqueName: \"kubernetes.io/projected/6c848b61-016a-46a2-8ce3-cb780d1d9f15-kube-api-access-5n25g\") pod \"noten-model-3edc13-kserve-7b975bbcdf-j8jbw\" (UID: \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\") " pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" Oct 30 19:33:34.166376 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.166269 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c848b61-016a-46a2-8ce3-cb780d1d9f15-tls-certs\") pod \"noten-model-3edc13-kserve-7b975bbcdf-j8jbw\" (UID: \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\") " pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" Oct 30 19:33:34.166376 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.166348 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-model-cache\") pod \"noten-model-3edc13-kserve-7b975bbcdf-j8jbw\" (UID: \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\") " pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" Oct 30 19:33:34.166479 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.166400 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-home\") pod \"noten-model-3edc13-kserve-7b975bbcdf-j8jbw\" (UID: \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\") " pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" Oct 30 19:33:34.166543 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.166524 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-tmp-dir\") pod \"noten-model-3edc13-kserve-7b975bbcdf-j8jbw\" (UID: \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\") " pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" Oct 30 19:33:34.166622 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.166604 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-dshm\") pod \"noten-model-3edc13-kserve-7b975bbcdf-j8jbw\" (UID: \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\") " pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" Oct 30 19:33:34.232212 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.232180 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-3edc13\"/\"default-dockercfg-mqrh7\"" Oct 30 19:33:34.232212 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.232181 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-3edc13\"/\"kube-root-ca.crt\"" Oct 30 19:33:34.232212 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.232183 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-3edc13\"/\"openshift-service-ca.crt\"" Oct 30 19:33:34.232472 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.232289 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-3edc13\"/\"noten-model-3edc13-kserve-self-signed-certs\"" Oct 30 19:33:34.235057 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.235034 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw"] Oct 30 19:33:34.268201 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.268081 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n25g\" (UniqueName: \"kubernetes.io/projected/6c848b61-016a-46a2-8ce3-cb780d1d9f15-kube-api-access-5n25g\") pod \"noten-model-3edc13-kserve-7b975bbcdf-j8jbw\" (UID: \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\") " pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" Oct 30 19:33:34.268201 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.268204 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c848b61-016a-46a2-8ce3-cb780d1d9f15-tls-certs\") pod \"noten-model-3edc13-kserve-7b975bbcdf-j8jbw\" (UID: \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\") " pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" Oct 30 19:33:34.268561 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.268232 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-model-cache\") pod \"noten-model-3edc13-kserve-7b975bbcdf-j8jbw\" (UID: \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\") " pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" Oct 30 19:33:34.268561 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.268272 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-home\") pod \"noten-model-3edc13-kserve-7b975bbcdf-j8jbw\" (UID: \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\") " pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" Oct 30 19:33:34.268561 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.268339 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-tmp-dir\") pod \"noten-model-3edc13-kserve-7b975bbcdf-j8jbw\" (UID: \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\") " pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" Oct 30 19:33:34.269112 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.269078 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-dshm\") pod \"noten-model-3edc13-kserve-7b975bbcdf-j8jbw\" (UID: \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\") " pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" Oct 30 19:33:34.273167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.269250 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-home\") pod \"noten-model-3edc13-kserve-7b975bbcdf-j8jbw\" (UID: \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\") " pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" Oct 30 19:33:34.273167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.269468 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-tmp-dir\") pod \"noten-model-3edc13-kserve-7b975bbcdf-j8jbw\" (UID: \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\") " pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" Oct 30 19:33:34.273167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.269530 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-model-cache\") pod \"noten-model-3edc13-kserve-7b975bbcdf-j8jbw\" (UID: \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\") " pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" Oct 30 19:33:34.273167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.271992 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c848b61-016a-46a2-8ce3-cb780d1d9f15-tls-certs\") pod \"noten-model-3edc13-kserve-7b975bbcdf-j8jbw\" (UID: \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\") " pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" Oct 30 19:33:34.286426 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.286384 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-dshm\") pod \"noten-model-3edc13-kserve-7b975bbcdf-j8jbw\" (UID: \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\") " pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" Oct 30 19:33:34.332425 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.332387 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n25g\" (UniqueName: \"kubernetes.io/projected/6c848b61-016a-46a2-8ce3-cb780d1d9f15-kube-api-access-5n25g\") pod \"noten-model-3edc13-kserve-7b975bbcdf-j8jbw\" (UID: \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\") " pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" Oct 30 19:33:34.443280 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.443175 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" Oct 30 19:33:34.834660 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:34.834634 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw"] Oct 30 19:33:34.836254 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:33:34.836229 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c848b61_016a_46a2_8ce3_cb780d1d9f15.slice/crio-f4cee65c6c1a05663a4f34bd21dabd60f9819ee783d5bbcbe1b47186d87e9e44 WatchSource:0}: Error finding container f4cee65c6c1a05663a4f34bd21dabd60f9819ee783d5bbcbe1b47186d87e9e44: Status 404 returned error can't find the container with id f4cee65c6c1a05663a4f34bd21dabd60f9819ee783d5bbcbe1b47186d87e9e44 Oct 30 19:33:35.155555 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:35.155461 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" event={"ID":"6c848b61-016a-46a2-8ce3-cb780d1d9f15","Type":"ContainerStarted","Data":"7b0bc2c345050bc0bd66a076621bbbcff71f51f225572b0dc5aa3ef9ca2ff41e"} Oct 30 19:33:35.155555 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:35.155508 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" event={"ID":"6c848b61-016a-46a2-8ce3-cb780d1d9f15","Type":"ContainerStarted","Data":"f4cee65c6c1a05663a4f34bd21dabd60f9819ee783d5bbcbe1b47186d87e9e44"} Oct 30 19:33:35.275074 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:35.275007 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" podStartSLOduration=1.2749925069999999 podStartE2EDuration="1.274992507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:35.272585035 +0000 UTC m=+3575.299689997" watchObservedRunningTime="2026-10-30 19:33:35.274992507 +0000 UTC m=+3575.302097467" Oct 30 19:33:35.443759 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:35.443681 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" Oct 30 19:33:40.126716 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:40.126684 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k" Oct 30 19:33:44.443668 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:44.443625 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" Oct 30 19:33:44.456817 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:44.456786 4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" Oct 30 19:33:44.776735 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:44.776690 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw"] Oct 30 19:33:45.228997 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:33:45.228892 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" Oct 30 19:33:45.298571 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:33:45.298524 4531 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-self-signed-certs: secret "noten-model-3edc13-kserve-self-signed-certs" not found Oct 30 19:33:45.298790 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:33:45.298609 4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c848b61-016a-46a2-8ce3-cb780d1d9f15-tls-certs podName:6c848b61-016a-46a2-8ce3-cb780d1d9f15 nodeName:}" failed. No retries permitted until 2026-10-30 19:33:45.798592284 +0000 UTC m=+3585.825697222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6c848b61-016a-46a2-8ce3-cb780d1d9f15-tls-certs") pod "noten-model-3edc13-kserve-7b975bbcdf-j8jbw" (UID: "6c848b61-016a-46a2-8ce3-cb780d1d9f15") : secret "noten-model-3edc13-kserve-self-signed-certs" not found Oct 30 19:33:45.803716 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:33:45.803676 4531 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-self-signed-certs: secret "noten-model-3edc13-kserve-self-signed-certs" not found Oct 30 19:33:45.804193 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:33:45.803760 4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c848b61-016a-46a2-8ce3-cb780d1d9f15-tls-certs podName:6c848b61-016a-46a2-8ce3-cb780d1d9f15 nodeName:}" failed. No retries permitted until 2026-10-30 19:33:46.803746003 +0000 UTC m=+3586.830850939 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6c848b61-016a-46a2-8ce3-cb780d1d9f15-tls-certs") pod "noten-model-3edc13-kserve-7b975bbcdf-j8jbw" (UID: "6c848b61-016a-46a2-8ce3-cb780d1d9f15") : secret "noten-model-3edc13-kserve-self-signed-certs" not found Oct 30 19:33:46.816218 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:33:46.816185 4531 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-self-signed-certs: secret "noten-model-3edc13-kserve-self-signed-certs" not found Oct 30 19:33:46.816642 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:33:46.816273 4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c848b61-016a-46a2-8ce3-cb780d1d9f15-tls-certs podName:6c848b61-016a-46a2-8ce3-cb780d1d9f15 nodeName:}" failed. No retries permitted until 2026-10-30 19:33:48.816251153 +0000 UTC m=+3588.843356103 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6c848b61-016a-46a2-8ce3-cb780d1d9f15-tls-certs") pod "noten-model-3edc13-kserve-7b975bbcdf-j8jbw" (UID: "6c848b61-016a-46a2-8ce3-cb780d1d9f15") : secret "noten-model-3edc13-kserve-self-signed-certs" not found Oct 30 19:33:48.839661 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:33:48.839623 4531 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-self-signed-certs: secret "noten-model-3edc13-kserve-self-signed-certs" not found Oct 30 19:33:48.840120 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:33:48.839714 4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c848b61-016a-46a2-8ce3-cb780d1d9f15-tls-certs podName:6c848b61-016a-46a2-8ce3-cb780d1d9f15 nodeName:}" failed. No retries permitted until 2026-10-30 19:33:52.839696824 +0000 UTC m=+3592.866801762 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6c848b61-016a-46a2-8ce3-cb780d1d9f15-tls-certs") pod "noten-model-3edc13-kserve-7b975bbcdf-j8jbw" (UID: "6c848b61-016a-46a2-8ce3-cb780d1d9f15") : secret "noten-model-3edc13-kserve-self-signed-certs" not found Oct 30 19:33:52.891558 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:33:52.891510 4531 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-self-signed-certs: secret "noten-model-3edc13-kserve-self-signed-certs" not found Oct 30 19:33:52.892198 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:33:52.891592 4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c848b61-016a-46a2-8ce3-cb780d1d9f15-tls-certs podName:6c848b61-016a-46a2-8ce3-cb780d1d9f15 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:00.891574115 +0000 UTC m=+3600.918679052 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6c848b61-016a-46a2-8ce3-cb780d1d9f15-tls-certs") pod "noten-model-3edc13-kserve-7b975bbcdf-j8jbw" (UID: "6c848b61-016a-46a2-8ce3-cb780d1d9f15") : secret "noten-model-3edc13-kserve-self-signed-certs" not found Oct 30 19:33:56.398760 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:33:56.398719 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:34:00.981833 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:34:00.981787 4531 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-self-signed-certs: secret "noten-model-3edc13-kserve-self-signed-certs" not found Oct 30 19:34:00.982339 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:34:00.981882 4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c848b61-016a-46a2-8ce3-cb780d1d9f15-tls-certs podName:6c848b61-016a-46a2-8ce3-cb780d1d9f15 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:16.981865526 +0000 UTC m=+3617.008970466 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6c848b61-016a-46a2-8ce3-cb780d1d9f15-tls-certs") pod "noten-model-3edc13-kserve-7b975bbcdf-j8jbw" (UID: "6c848b61-016a-46a2-8ce3-cb780d1d9f15") : secret "noten-model-3edc13-kserve-self-signed-certs" not found Oct 30 19:34:01.244270 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:01.244207 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" podUID="6c848b61-016a-46a2-8ce3-cb780d1d9f15" containerName="main" containerID="cri-o://7b0bc2c345050bc0bd66a076621bbbcff71f51f225572b0dc5aa3ef9ca2ff41e" gracePeriod=45 Oct 30 19:34:01.523822 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:01.523789 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" Oct 30 19:34:01.589563 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:01.589516 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-dshm\") pod \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\" (UID: \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\") " Oct 30 19:34:01.589731 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:01.589586 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c848b61-016a-46a2-8ce3-cb780d1d9f15-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c848b61-016a-46a2-8ce3-cb780d1d9f15-tls-certs\") pod \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\" (UID: \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\") " Oct 30 19:34:01.589731 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:01.589715 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-model-cache\") pod \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\" (UID: \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\") " Oct 30 19:34:01.589846 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:01.589765 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-tmp-dir\") pod \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\" (UID: \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\") " Oct 30 19:34:01.589846 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:01.589835 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-home\" (UniqueName: \"kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-home\") pod \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\" (UID: \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\") " Oct 30 19:34:01.589939 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:01.589909 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c848b61-016a-46a2-8ce3-cb780d1d9f15-kube-api-access-5n25g\" (UniqueName: \"kubernetes.io/projected/6c848b61-016a-46a2-8ce3-cb780d1d9f15-kube-api-access-5n25g\") pod \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\" (UID: \"6c848b61-016a-46a2-8ce3-cb780d1d9f15\") " Oct 30 19:34:01.590063 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:01.590012 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-model-cache" pod "6c848b61-016a-46a2-8ce3-cb780d1d9f15" (UID: "6c848b61-016a-46a2-8ce3-cb780d1d9f15"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:01.590063 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:01.590058 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-tmp-dir" pod "6c848b61-016a-46a2-8ce3-cb780d1d9f15" (UID: "6c848b61-016a-46a2-8ce3-cb780d1d9f15"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:01.590342 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:01.590259 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-home" pod "6c848b61-016a-46a2-8ce3-cb780d1d9f15" (UID: "6c848b61-016a-46a2-8ce3-cb780d1d9f15"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:01.590624 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:01.590601 4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-model-cache\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:34:01.590889 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:01.590629 4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-tmp-dir\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:34:01.590889 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:01.590647 4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-home\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:34:01.592231 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:01.592203 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c848b61-016a-46a2-8ce3-cb780d1d9f15-tls-certs" pod "6c848b61-016a-46a2-8ce3-cb780d1d9f15" (UID: "6c848b61-016a-46a2-8ce3-cb780d1d9f15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:34:01.592360 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:01.592336 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-dshm" pod "6c848b61-016a-46a2-8ce3-cb780d1d9f15" (UID: "6c848b61-016a-46a2-8ce3-cb780d1d9f15"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:01.592406 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:01.592361 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c848b61-016a-46a2-8ce3-cb780d1d9f15-kube-api-access-5n25g" pod "6c848b61-016a-46a2-8ce3-cb780d1d9f15" (UID: "6c848b61-016a-46a2-8ce3-cb780d1d9f15"). InnerVolumeSpecName "kube-api-access-5n25g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:01.692291 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:01.692247 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n25g\" (UniqueName: \"kubernetes.io/projected/6c848b61-016a-46a2-8ce3-cb780d1d9f15-kube-api-access-5n25g\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:34:01.692291 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:01.692279 4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c848b61-016a-46a2-8ce3-cb780d1d9f15-dshm\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:34:01.692291 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:01.692290 4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c848b61-016a-46a2-8ce3-cb780d1d9f15-tls-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:34:02.323843 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:02.323801 4531 generic.go:356] "Generic (PLEG): container finished" podID="6c848b61-016a-46a2-8ce3-cb780d1d9f15" containerID="7b0bc2c345050bc0bd66a076621bbbcff71f51f225572b0dc5aa3ef9ca2ff41e" exitCode=0 Oct 30 19:34:02.324448 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:02.323872 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" Oct 30 19:34:02.324448 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:02.323889 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" event={"ID":"6c848b61-016a-46a2-8ce3-cb780d1d9f15","Type":"ContainerDied","Data":"7b0bc2c345050bc0bd66a076621bbbcff71f51f225572b0dc5aa3ef9ca2ff41e"} Oct 30 19:34:02.324448 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:02.323945 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw" event={"ID":"6c848b61-016a-46a2-8ce3-cb780d1d9f15","Type":"ContainerDied","Data":"f4cee65c6c1a05663a4f34bd21dabd60f9819ee783d5bbcbe1b47186d87e9e44"} Oct 30 19:34:02.324448 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:02.323971 4531 scope.go:122] "RemoveContainer" containerID="7b0bc2c345050bc0bd66a076621bbbcff71f51f225572b0dc5aa3ef9ca2ff41e" Oct 30 19:34:02.336324 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:02.336298 4531 scope.go:122] "RemoveContainer" containerID="7b0bc2c345050bc0bd66a076621bbbcff71f51f225572b0dc5aa3ef9ca2ff41e" Oct 30 19:34:02.336747 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:34:02.336726 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b0bc2c345050bc0bd66a076621bbbcff71f51f225572b0dc5aa3ef9ca2ff41e\": container with ID starting with 7b0bc2c345050bc0bd66a076621bbbcff71f51f225572b0dc5aa3ef9ca2ff41e not found: ID does not exist" containerID="7b0bc2c345050bc0bd66a076621bbbcff71f51f225572b0dc5aa3ef9ca2ff41e" Oct 30 19:34:02.336807 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:02.336757 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b0bc2c345050bc0bd66a076621bbbcff71f51f225572b0dc5aa3ef9ca2ff41e"} err="failed to get container status \"7b0bc2c345050bc0bd66a076621bbbcff71f51f225572b0dc5aa3ef9ca2ff41e\": rpc error: code = NotFound desc = could not find container \"7b0bc2c345050bc0bd66a076621bbbcff71f51f225572b0dc5aa3ef9ca2ff41e\": container with ID starting with 7b0bc2c345050bc0bd66a076621bbbcff71f51f225572b0dc5aa3ef9ca2ff41e not found: ID does not exist" Oct 30 19:34:02.489358 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:02.489262 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw"] Oct 30 19:34:02.539230 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:02.539196 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-3edc13/noten-model-3edc13-kserve-7b975bbcdf-j8jbw"] Oct 30 19:34:02.953020 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:02.952988 4531 scope.go:122] "RemoveContainer" containerID="1576a4e547ae1cf82e5adf180c6bc939c7d4e895b182aca0a98fd6978ebb8d33" Oct 30 19:34:02.963083 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:02.963050 4531 scope.go:122] "RemoveContainer" containerID="e82d8e853cde3d8a69668f7ddaff6fd2bc5b0d99ab2320ffff549b0e9b655f30" Oct 30 19:34:04.099396 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:04.099353 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c848b61-016a-46a2-8ce3-cb780d1d9f15" path="/var/lib/kubelet/pods/6c848b61-016a-46a2-8ce3-cb780d1d9f15/volumes" Oct 30 19:34:22.050593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:22.050558 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55b37bbe-gs8lv"] Oct 30 19:34:22.100157 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:22.100097 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55b37bbe-gs8lv"] Oct 30 19:34:24.106487 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:24.106441 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a139411-416b-4beb-b089-fe85e87b485e" path="/var/lib/kubelet/pods/9a139411-416b-4beb-b089-fe85e87b485e/volumes" Oct 30 19:34:26.399871 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:34:26.399830 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:34:33.101057 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:33.100987 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd1b3d8a-vr8tr"] Oct 30 19:34:33.150406 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:33.150364 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd1b3d8a-vr8tr"] Oct 30 19:34:34.104304 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.104258 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edde5fa1-8c74-4755-b931-895636099038" path="/var/lib/kubelet/pods/edde5fa1-8c74-4755-b931-895636099038/volumes" Oct 30 19:34:34.214891 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.214855 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs"] Oct 30 19:34:34.215398 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.215382 4531 cpu_manager.go:414] "containerMap: removing container" podUID="6c848b61-016a-46a2-8ce3-cb780d1d9f15" containerName="main" Oct 30 19:34:34.215398 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.215397 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c848b61-016a-46a2-8ce3-cb780d1d9f15" containerName="main" Oct 30 19:34:34.215552 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.215493 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c848b61-016a-46a2-8ce3-cb780d1d9f15" containerName="main" Oct 30 19:34:34.218870 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.218844 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs" Oct 30 19:34:34.271280 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.271238 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-76a1ef\"" Oct 30 19:34:34.288650 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.288598 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f38975-1b9b-47d0-93c1-0ab25c13d127-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs\" (UID: \"d9f38975-1b9b-47d0-93c1-0ab25c13d127\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs" Oct 30 19:34:34.288841 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.288806 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvhmt\" (UniqueName: \"kubernetes.io/projected/d9f38975-1b9b-47d0-93c1-0ab25c13d127-kube-api-access-bvhmt\") pod \"payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs\" (UID: \"d9f38975-1b9b-47d0-93c1-0ab25c13d127\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs" Oct 30 19:34:34.320728 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.320689 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-76a1ef-dockercfg-5rdz9\"" Oct 30 19:34:34.324962 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.324916 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs"] Oct 30 19:34:34.374237 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.374149 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m"] Oct 30 19:34:34.378099 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.378073 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m" Oct 30 19:34:34.389639 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.389612 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f38975-1b9b-47d0-93c1-0ab25c13d127-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs\" (UID: \"d9f38975-1b9b-47d0-93c1-0ab25c13d127\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs" Oct 30 19:34:34.389804 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.389687 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvhmt\" (UniqueName: \"kubernetes.io/projected/d9f38975-1b9b-47d0-93c1-0ab25c13d127-kube-api-access-bvhmt\") pod \"payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs\" (UID: \"d9f38975-1b9b-47d0-93c1-0ab25c13d127\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs" Oct 30 19:34:34.390364 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.390339 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f38975-1b9b-47d0-93c1-0ab25c13d127-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs\" (UID: \"d9f38975-1b9b-47d0-93c1-0ab25c13d127\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs" Oct 30 19:34:34.456558 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.456522 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m"] Oct 30 19:34:34.465495 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.465456 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvhmt\" (UniqueName: \"kubernetes.io/projected/d9f38975-1b9b-47d0-93c1-0ab25c13d127-kube-api-access-bvhmt\") pod \"payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs\" (UID: \"d9f38975-1b9b-47d0-93c1-0ab25c13d127\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs" Oct 30 19:34:34.490749 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.490705 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387d077d-6792-45c0-ae7c-2891c00f47c2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m\" (UID: \"387d077d-6792-45c0-ae7c-2891c00f47c2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m" Oct 30 19:34:34.490749 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.490755 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxxn7\" (UniqueName: \"kubernetes.io/projected/387d077d-6792-45c0-ae7c-2891c00f47c2-kube-api-access-mxxn7\") pod \"payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m\" (UID: \"387d077d-6792-45c0-ae7c-2891c00f47c2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m" Oct 30 19:34:34.533362 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.533323 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs" Oct 30 19:34:34.592912 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.592876 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387d077d-6792-45c0-ae7c-2891c00f47c2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m\" (UID: \"387d077d-6792-45c0-ae7c-2891c00f47c2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m" Oct 30 19:34:34.593035 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.592949 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxxn7\" (UniqueName: \"kubernetes.io/projected/387d077d-6792-45c0-ae7c-2891c00f47c2-kube-api-access-mxxn7\") pod \"payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m\" (UID: \"387d077d-6792-45c0-ae7c-2891c00f47c2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m" Oct 30 19:34:34.594541 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.594504 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387d077d-6792-45c0-ae7c-2891c00f47c2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m\" (UID: \"387d077d-6792-45c0-ae7c-2891c00f47c2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m" Oct 30 19:34:34.655268 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.655181 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxxn7\" (UniqueName: \"kubernetes.io/projected/387d077d-6792-45c0-ae7c-2891c00f47c2-kube-api-access-mxxn7\") pod \"payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m\" (UID: \"387d077d-6792-45c0-ae7c-2891c00f47c2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m" Oct 30 19:34:34.692525 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.692489 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m" Oct 30 19:34:34.914345 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:34.914318 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs"] Oct 30 19:34:34.916296 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:34:34.916263 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f38975_1b9b_47d0_93c1_0ab25c13d127.slice/crio-0cdf548ab478fc43fe9f2e67ec6b922fe2ff7300f3af690f39edadf90bdd3f5b WatchSource:0}: Error finding container 0cdf548ab478fc43fe9f2e67ec6b922fe2ff7300f3af690f39edadf90bdd3f5b: Status 404 returned error can't find the container with id 0cdf548ab478fc43fe9f2e67ec6b922fe2ff7300f3af690f39edadf90bdd3f5b Oct 30 19:34:35.071839 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:35.071809 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m"] Oct 30 19:34:35.072201 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:34:35.072176 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod387d077d_6792_45c0_ae7c_2891c00f47c2.slice/crio-79898ad9ce8a5f6e11340684b37c506ba28534ccb6b589929c9b2ca3039d3cd2 WatchSource:0}: Error finding container 79898ad9ce8a5f6e11340684b37c506ba28534ccb6b589929c9b2ca3039d3cd2: Status 404 returned error can't find the container with id 79898ad9ce8a5f6e11340684b37c506ba28534ccb6b589929c9b2ca3039d3cd2 Oct 30 19:34:35.534650 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:35.534611 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs" event={"ID":"d9f38975-1b9b-47d0-93c1-0ab25c13d127","Type":"ContainerStarted","Data":"4db96304fc3ebc271636b13adcdcd46301d9642c20755393c8c89098c14a4f97"} Oct 30 19:34:35.535152 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:35.534682 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs" Oct 30 19:34:35.535152 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:35.534700 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs" event={"ID":"d9f38975-1b9b-47d0-93c1-0ab25c13d127","Type":"ContainerStarted","Data":"0cdf548ab478fc43fe9f2e67ec6b922fe2ff7300f3af690f39edadf90bdd3f5b"} Oct 30 19:34:35.536596 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:35.536559 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m" event={"ID":"387d077d-6792-45c0-ae7c-2891c00f47c2","Type":"ContainerStarted","Data":"9186cfce414a0a851a046cf9da2260bfa8a94b50215af55b550cf564ad729866"} Oct 30 19:34:35.536768 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:35.536616 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m" event={"ID":"387d077d-6792-45c0-ae7c-2891c00f47c2","Type":"ContainerStarted","Data":"79898ad9ce8a5f6e11340684b37c506ba28534ccb6b589929c9b2ca3039d3cd2"} Oct 30 19:34:35.536768 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:35.536764 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m" Oct 30 19:34:35.652227 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:35.652172 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs" podStartSLOduration=1.652157416 podStartE2EDuration="1.652157416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:35.649696086 +0000 UTC m=+3635.676801044" watchObservedRunningTime="2026-10-30 19:34:35.652157416 +0000 UTC m=+3635.679262371" Oct 30 19:34:35.767763 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:35.767695 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m" podStartSLOduration=1.767679465 podStartE2EDuration="1.767679465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:35.766321053 +0000 UTC m=+3635.793426010" watchObservedRunningTime="2026-10-30 19:34:35.767679465 +0000 UTC m=+3635.794784423" Oct 30 19:34:41.606370 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:41.606284 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4"] Oct 30 19:34:41.610119 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:41.610103 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4" Oct 30 19:34:41.672475 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:41.672435 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89fa9ca7-786c-4d84-abce-46b6e81f87bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4\" (UID: \"89fa9ca7-786c-4d84-abce-46b6e81f87bc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4" Oct 30 19:34:41.672885 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:41.672860 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh4gl\" (UniqueName: \"kubernetes.io/projected/89fa9ca7-786c-4d84-abce-46b6e81f87bc-kube-api-access-kh4gl\") pod \"payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4\" (UID: \"89fa9ca7-786c-4d84-abce-46b6e81f87bc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4" Oct 30 19:34:41.711365 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:41.711327 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-25e5ef\"" Oct 30 19:34:41.711557 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:41.711460 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-25e5ef-dockercfg-5zmqw\"" Oct 30 19:34:41.716412 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:41.716380 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4"] Oct 30 19:34:41.764464 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:41.764430 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv"] Oct 30 19:34:41.768464 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:41.768435 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv" Oct 30 19:34:41.773850 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:41.773815 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh4gl\" (UniqueName: \"kubernetes.io/projected/89fa9ca7-786c-4d84-abce-46b6e81f87bc-kube-api-access-kh4gl\") pod \"payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4\" (UID: \"89fa9ca7-786c-4d84-abce-46b6e81f87bc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4" Oct 30 19:34:41.774042 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:41.773930 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89fa9ca7-786c-4d84-abce-46b6e81f87bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4\" (UID: \"89fa9ca7-786c-4d84-abce-46b6e81f87bc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4" Oct 30 19:34:41.774821 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:41.774800 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89fa9ca7-786c-4d84-abce-46b6e81f87bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4\" (UID: \"89fa9ca7-786c-4d84-abce-46b6e81f87bc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4" Oct 30 19:34:41.809522 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:41.809481 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv"] Oct 30 19:34:41.835586 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:41.835552 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh4gl\" (UniqueName: \"kubernetes.io/projected/89fa9ca7-786c-4d84-abce-46b6e81f87bc-kube-api-access-kh4gl\") pod \"payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4\" (UID: \"89fa9ca7-786c-4d84-abce-46b6e81f87bc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4" Oct 30 19:34:41.874724 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:41.874619 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd72v\" (UniqueName: \"kubernetes.io/projected/47e3267f-ce2e-4d73-b3d6-ac88d5316300-kube-api-access-sd72v\") pod \"payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv\" (UID: \"47e3267f-ce2e-4d73-b3d6-ac88d5316300\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv" Oct 30 19:34:41.874915 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:41.874783 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e3267f-ce2e-4d73-b3d6-ac88d5316300-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv\" (UID: \"47e3267f-ce2e-4d73-b3d6-ac88d5316300\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv" Oct 30 19:34:41.924379 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:41.924334 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4" Oct 30 19:34:41.976295 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:41.976250 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd72v\" (UniqueName: \"kubernetes.io/projected/47e3267f-ce2e-4d73-b3d6-ac88d5316300-kube-api-access-sd72v\") pod \"payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv\" (UID: \"47e3267f-ce2e-4d73-b3d6-ac88d5316300\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv" Oct 30 19:34:41.976482 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:41.976372 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e3267f-ce2e-4d73-b3d6-ac88d5316300-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv\" (UID: \"47e3267f-ce2e-4d73-b3d6-ac88d5316300\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv" Oct 30 19:34:41.977173 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:41.977151 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e3267f-ce2e-4d73-b3d6-ac88d5316300-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv\" (UID: \"47e3267f-ce2e-4d73-b3d6-ac88d5316300\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv" Oct 30 19:34:42.038540 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:42.038505 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd72v\" (UniqueName: \"kubernetes.io/projected/47e3267f-ce2e-4d73-b3d6-ac88d5316300-kube-api-access-sd72v\") pod \"payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv\" (UID: \"47e3267f-ce2e-4d73-b3d6-ac88d5316300\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv" Oct 30 19:34:42.084469 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:42.084439 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv" Oct 30 19:34:42.310032 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:42.309996 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4"] Oct 30 19:34:42.312278 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:34:42.312251 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89fa9ca7_786c_4d84_abce_46b6e81f87bc.slice/crio-509418ef264c110d8fe0f189b45bf8009a132a5cebcd4cb3fcec17b830c6fb90 WatchSource:0}: Error finding container 509418ef264c110d8fe0f189b45bf8009a132a5cebcd4cb3fcec17b830c6fb90: Status 404 returned error can't find the container with id 509418ef264c110d8fe0f189b45bf8009a132a5cebcd4cb3fcec17b830c6fb90 Oct 30 19:34:42.462482 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:42.462456 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv"] Oct 30 19:34:42.464055 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:34:42.464030 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e3267f_ce2e_4d73_b3d6_ac88d5316300.slice/crio-f39092dcccbb9594fb5d1b00c2db9006c564bc09408fd4d313113649674d1e0c WatchSource:0}: Error finding container f39092dcccbb9594fb5d1b00c2db9006c564bc09408fd4d313113649674d1e0c: Status 404 returned error can't find the container with id f39092dcccbb9594fb5d1b00c2db9006c564bc09408fd4d313113649674d1e0c Oct 30 19:34:42.580424 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:42.580381 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv" event={"ID":"47e3267f-ce2e-4d73-b3d6-ac88d5316300","Type":"ContainerStarted","Data":"59a54abfea72103028ba6a69d012e9717fd25c092683c5a3ed51922b7ba49298"} Oct 30 19:34:42.580612 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:42.580454 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv" event={"ID":"47e3267f-ce2e-4d73-b3d6-ac88d5316300","Type":"ContainerStarted","Data":"f39092dcccbb9594fb5d1b00c2db9006c564bc09408fd4d313113649674d1e0c"} Oct 30 19:34:42.580612 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:42.580481 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv" Oct 30 19:34:42.582305 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:42.582276 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4" event={"ID":"89fa9ca7-786c-4d84-abce-46b6e81f87bc","Type":"ContainerStarted","Data":"fce2c87a6445ddcebeee09fc921c9efd077f3bd92727e321b07476143194e22d"} Oct 30 19:34:42.582437 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:42.582328 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4" event={"ID":"89fa9ca7-786c-4d84-abce-46b6e81f87bc","Type":"ContainerStarted","Data":"509418ef264c110d8fe0f189b45bf8009a132a5cebcd4cb3fcec17b830c6fb90"} Oct 30 19:34:42.582437 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:42.582402 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4" Oct 30 19:34:42.700737 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:42.700680 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv" podStartSLOduration=1.7006622980000001 podStartE2EDuration="1.700662298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:42.697549704 +0000 UTC m=+3642.724654662" watchObservedRunningTime="2026-10-30 19:34:42.700662298 +0000 UTC m=+3642.727767264" Oct 30 19:34:42.816928 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:42.816819 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4" podStartSLOduration=1.816798513 podStartE2EDuration="1.816798513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:42.813483452 +0000 UTC m=+3642.840588413" watchObservedRunningTime="2026-10-30 19:34:42.816798513 +0000 UTC m=+3642.843903469" Oct 30 19:34:46.547386 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:46.547351 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m" Oct 30 19:34:46.547901 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:46.547433 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs" Oct 30 19:34:53.593408 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:53.593376 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv" Oct 30 19:34:53.593856 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:34:53.593463 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4" Oct 30 19:34:56.399509 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:34:56.399471 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:35:26.401787 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:35:26.401739 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:35:38.060554 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:35:38.060511 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a2e084f-bbs2x"] Oct 30 19:35:38.108618 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:35:38.108565 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a2e084f-bbs2x"] Oct 30 19:35:40.100022 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:35:40.099985 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cf097bb-7649-46a5-a1a7-ba3359454574" path="/var/lib/kubelet/pods/6cf097bb-7649-46a5-a1a7-ba3359454574/volumes" Oct 30 19:35:45.527675 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:35:45.527628 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g"] Oct 30 19:35:45.535962 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:35:45.535925 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g" Oct 30 19:35:45.591211 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:35:45.590811 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g"] Oct 30 19:35:45.636259 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:35:45.636228 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-345670-dockercfg-g5qjx\"" Oct 30 19:35:45.636259 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:35:45.636245 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-345670\"" Oct 30 19:35:45.719105 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:35:45.719067 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx5dn\" (UniqueName: \"kubernetes.io/projected/e9b41342-7ece-47b2-bb04-129a141a0a2d-kube-api-access-mx5dn\") pod \"payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g\" (UID: \"e9b41342-7ece-47b2-bb04-129a141a0a2d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g" Oct 30 19:35:45.719312 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:35:45.719257 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9b41342-7ece-47b2-bb04-129a141a0a2d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g\" (UID: \"e9b41342-7ece-47b2-bb04-129a141a0a2d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g" Oct 30 19:35:45.820050 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:35:45.819940 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx5dn\" (UniqueName: \"kubernetes.io/projected/e9b41342-7ece-47b2-bb04-129a141a0a2d-kube-api-access-mx5dn\") pod \"payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g\" (UID: \"e9b41342-7ece-47b2-bb04-129a141a0a2d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g" Oct 30 19:35:45.820293 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:35:45.820081 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9b41342-7ece-47b2-bb04-129a141a0a2d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g\" (UID: \"e9b41342-7ece-47b2-bb04-129a141a0a2d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g" Oct 30 19:35:45.820893 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:35:45.820872 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9b41342-7ece-47b2-bb04-129a141a0a2d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g\" (UID: \"e9b41342-7ece-47b2-bb04-129a141a0a2d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g" Oct 30 19:35:45.882842 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:35:45.882806 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx5dn\" (UniqueName: \"kubernetes.io/projected/e9b41342-7ece-47b2-bb04-129a141a0a2d-kube-api-access-mx5dn\") pod \"payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g\" (UID: \"e9b41342-7ece-47b2-bb04-129a141a0a2d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g" Oct 30 19:35:46.152895 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:35:46.152799 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g" Oct 30 19:35:46.538206 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:35:46.538176 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g"] Oct 30 19:35:46.540425 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:35:46.540393 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9b41342_7ece_47b2_bb04_129a141a0a2d.slice/crio-d99a55507018aeb8a857bf277ba9b1b38623ee1453e4e60ca572f94dd778cc74 WatchSource:0}: Error finding container d99a55507018aeb8a857bf277ba9b1b38623ee1453e4e60ca572f94dd778cc74: Status 404 returned error can't find the container with id d99a55507018aeb8a857bf277ba9b1b38623ee1453e4e60ca572f94dd778cc74 Oct 30 19:35:46.993971 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:35:46.993935 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g" event={"ID":"e9b41342-7ece-47b2-bb04-129a141a0a2d","Type":"ContainerStarted","Data":"2ba0ef5b35515278e9731068a43b0913263293c6ccf5b900a2eec38cd2d5dfe5"} Oct 30 19:35:46.994199 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:35:46.993990 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g" event={"ID":"e9b41342-7ece-47b2-bb04-129a141a0a2d","Type":"ContainerStarted","Data":"d99a55507018aeb8a857bf277ba9b1b38623ee1453e4e60ca572f94dd778cc74"} Oct 30 19:35:46.994199 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:35:46.994156 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g" Oct 30 19:35:47.115476 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:35:47.115411 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g" podStartSLOduration=2.115393872 podStartE2EDuration="2.115393872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:47.113043344 +0000 UTC m=+3707.140148305" watchObservedRunningTime="2026-10-30 19:35:47.115393872 +0000 UTC m=+3707.142498831" Oct 30 19:35:49.053556 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:35:49.053508 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8d65af8-g68qs"] Oct 30 19:35:49.102939 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:35:49.102901 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8d65af8-g68qs"] Oct 30 19:35:50.100423 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:35:50.100386 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96d9b8d4-5464-4f47-984f-2af77126dede" path="/var/lib/kubelet/pods/96d9b8d4-5464-4f47-984f-2af77126dede/volumes" Oct 30 19:35:56.399880 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:35:56.399842 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:35:58.004266 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:35:58.004236 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g" Oct 30 19:36:03.007601 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:03.007566 4531 scope.go:122] "RemoveContainer" containerID="1b753196aef6f0182586f15a5c1f9e6b02b2b0479722702c41b7eb90614b0162" Oct 30 19:36:03.018194 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:03.018164 4531 scope.go:122] "RemoveContainer" containerID="1e9720ea88cce2c49a8eeab5260b93ae522dffdf4a97f45b530c821934d51b65" Oct 30 19:36:03.028552 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:03.028527 4531 scope.go:122] "RemoveContainer" containerID="a0775ecc3db7edbd332d094362e6a568668b157cd700620136269a9b47af1ee9" Oct 30 19:36:10.795697 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:10.795654 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs"] Oct 30 19:36:10.799734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:10.799710 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs" Oct 30 19:36:10.858494 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:10.858456 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:36:10.908001 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:10.907911 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:36:10.908001 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:10.907967 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:36:10.908001 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:10.907973 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:36:10.908324 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:10.908302 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7k8v\" (UniqueName: \"kubernetes.io/projected/7a078ee1-bc3f-4e8f-8a10-e319d18c3030-kube-api-access-h7k8v\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs\" (UID: \"7a078ee1-bc3f-4e8f-8a10-e319d18c3030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs" Oct 30 19:36:10.908370 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:10.908343 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a078ee1-bc3f-4e8f-8a10-e319d18c3030-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs\" (UID: \"7a078ee1-bc3f-4e8f-8a10-e319d18c3030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs" Oct 30 19:36:10.908571 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:10.908542 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a078ee1-bc3f-4e8f-8a10-e319d18c3030-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs\" (UID: \"7a078ee1-bc3f-4e8f-8a10-e319d18c3030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs" Oct 30 19:36:10.911043 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:10.911012 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs"] Oct 30 19:36:11.010263 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:11.010218 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7k8v\" (UniqueName: \"kubernetes.io/projected/7a078ee1-bc3f-4e8f-8a10-e319d18c3030-kube-api-access-h7k8v\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs\" (UID: \"7a078ee1-bc3f-4e8f-8a10-e319d18c3030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs" Oct 30 19:36:11.010263 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:11.010269 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a078ee1-bc3f-4e8f-8a10-e319d18c3030-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs\" (UID: \"7a078ee1-bc3f-4e8f-8a10-e319d18c3030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs" Oct 30 19:36:11.010523 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:11.010357 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a078ee1-bc3f-4e8f-8a10-e319d18c3030-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs\" (UID: \"7a078ee1-bc3f-4e8f-8a10-e319d18c3030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs" Oct 30 19:36:11.011050 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:11.011025 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a078ee1-bc3f-4e8f-8a10-e319d18c3030-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs\" (UID: \"7a078ee1-bc3f-4e8f-8a10-e319d18c3030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs" Oct 30 19:36:11.012981 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:11.012949 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a078ee1-bc3f-4e8f-8a10-e319d18c3030-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs\" (UID: \"7a078ee1-bc3f-4e8f-8a10-e319d18c3030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs" Oct 30 19:36:11.072756 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:11.072672 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7k8v\" (UniqueName: \"kubernetes.io/projected/7a078ee1-bc3f-4e8f-8a10-e319d18c3030-kube-api-access-h7k8v\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs\" (UID: \"7a078ee1-bc3f-4e8f-8a10-e319d18c3030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs" Oct 30 19:36:11.114786 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:11.114739 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs" Oct 30 19:36:11.512324 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:11.512286 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs"] Oct 30 19:36:11.514169 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:36:11.514116 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a078ee1_bc3f_4e8f_8a10_e319d18c3030.slice/crio-1a65bbd9f51e8aab406b70985008cfef731b44d8d1c4c11f276dc15e8e090ae2 WatchSource:0}: Error finding container 1a65bbd9f51e8aab406b70985008cfef731b44d8d1c4c11f276dc15e8e090ae2: Status 404 returned error can't find the container with id 1a65bbd9f51e8aab406b70985008cfef731b44d8d1c4c11f276dc15e8e090ae2 Oct 30 19:36:12.156249 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:12.156099 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs" event={"ID":"7a078ee1-bc3f-4e8f-8a10-e319d18c3030","Type":"ContainerStarted","Data":"fa645bc3bdfd2896836bb42e73ed845cbe6a236a785599e7425c02a2df3d0187"} Oct 30 19:36:12.156249 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:12.156210 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs" event={"ID":"7a078ee1-bc3f-4e8f-8a10-e319d18c3030","Type":"ContainerStarted","Data":"1a65bbd9f51e8aab406b70985008cfef731b44d8d1c4c11f276dc15e8e090ae2"} Oct 30 19:36:12.274612 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:12.274556 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs" podStartSLOduration=2.274536795 podStartE2EDuration="2.274536795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:12.2713915 +0000 UTC m=+3732.298496465" watchObservedRunningTime="2026-10-30 19:36:12.274536795 +0000 UTC m=+3732.301641756" Oct 30 19:36:13.165056 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:13.165016 4531 generic.go:356] "Generic (PLEG): container finished" podID="7a078ee1-bc3f-4e8f-8a10-e319d18c3030" containerID="fa645bc3bdfd2896836bb42e73ed845cbe6a236a785599e7425c02a2df3d0187" exitCode=0 Oct 30 19:36:13.165526 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:13.165109 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs" event={"ID":"7a078ee1-bc3f-4e8f-8a10-e319d18c3030","Type":"ContainerDied","Data":"fa645bc3bdfd2896836bb42e73ed845cbe6a236a785599e7425c02a2df3d0187"} Oct 30 19:36:14.057877 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:14.057838 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cac4d782-sps87"] Oct 30 19:36:14.106545 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:14.106509 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cac4d782-sps87"] Oct 30 19:36:14.339623 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:14.339586 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs" Oct 30 19:36:14.456075 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:14.456037 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a078ee1-bc3f-4e8f-8a10-e319d18c3030-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a078ee1-bc3f-4e8f-8a10-e319d18c3030-maas-api-xks-ca\") pod \"7a078ee1-bc3f-4e8f-8a10-e319d18c3030\" (UID: \"7a078ee1-bc3f-4e8f-8a10-e319d18c3030\") " Oct 30 19:36:14.456304 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:14.456183 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a078ee1-bc3f-4e8f-8a10-e319d18c3030-kube-api-access-h7k8v\" (UniqueName: \"kubernetes.io/projected/7a078ee1-bc3f-4e8f-8a10-e319d18c3030-kube-api-access-h7k8v\") pod \"7a078ee1-bc3f-4e8f-8a10-e319d18c3030\" (UID: \"7a078ee1-bc3f-4e8f-8a10-e319d18c3030\") " Oct 30 19:36:14.456304 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:14.456275 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a078ee1-bc3f-4e8f-8a10-e319d18c3030-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a078ee1-bc3f-4e8f-8a10-e319d18c3030-maas-api-openshift-ca\") pod \"7a078ee1-bc3f-4e8f-8a10-e319d18c3030\" (UID: \"7a078ee1-bc3f-4e8f-8a10-e319d18c3030\") " Oct 30 19:36:14.456650 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:14.456624 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a078ee1-bc3f-4e8f-8a10-e319d18c3030-maas-api-openshift-ca" pod "7a078ee1-bc3f-4e8f-8a10-e319d18c3030" (UID: "7a078ee1-bc3f-4e8f-8a10-e319d18c3030"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:14.458571 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:14.458542 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a078ee1-bc3f-4e8f-8a10-e319d18c3030-maas-api-xks-ca" pod "7a078ee1-bc3f-4e8f-8a10-e319d18c3030" (UID: "7a078ee1-bc3f-4e8f-8a10-e319d18c3030"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:36:14.458571 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:14.458553 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a078ee1-bc3f-4e8f-8a10-e319d18c3030-kube-api-access-h7k8v" pod "7a078ee1-bc3f-4e8f-8a10-e319d18c3030" (UID: "7a078ee1-bc3f-4e8f-8a10-e319d18c3030"). InnerVolumeSpecName "kube-api-access-h7k8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:14.558095 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:14.558055 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7k8v\" (UniqueName: \"kubernetes.io/projected/7a078ee1-bc3f-4e8f-8a10-e319d18c3030-kube-api-access-h7k8v\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:36:14.558095 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:14.558088 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a078ee1-bc3f-4e8f-8a10-e319d18c3030-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:36:14.558095 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:14.558100 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a078ee1-bc3f-4e8f-8a10-e319d18c3030-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:36:15.179931 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:15.179905 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs" Oct 30 19:36:15.180109 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:15.179898 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs" event={"ID":"7a078ee1-bc3f-4e8f-8a10-e319d18c3030","Type":"ContainerDied","Data":"1a65bbd9f51e8aab406b70985008cfef731b44d8d1c4c11f276dc15e8e090ae2"} Oct 30 19:36:15.180109 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:15.180031 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a65bbd9f51e8aab406b70985008cfef731b44d8d1c4c11f276dc15e8e090ae2" Oct 30 19:36:16.099612 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:16.099572 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5a092e5-a230-4564-b8f3-aeabf5580acc" path="/var/lib/kubelet/pods/a5a092e5-a230-4564-b8f3-aeabf5580acc/volumes" Oct 30 19:36:22.295658 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:22.295611 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg"] Oct 30 19:36:22.405935 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:22.405883 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v"] Oct 30 19:36:23.061779 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:23.061734 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd"] Oct 30 19:36:23.062357 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:23.062341 4531 cpu_manager.go:414] "containerMap: removing container" podUID="7a078ee1-bc3f-4e8f-8a10-e319d18c3030" containerName="revoke-keys" Oct 30 19:36:23.062428 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:23.062358 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a078ee1-bc3f-4e8f-8a10-e319d18c3030" containerName="revoke-keys" Oct 30 19:36:23.062471 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:23.062455 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a078ee1-bc3f-4e8f-8a10-e319d18c3030" containerName="revoke-keys" Oct 30 19:36:23.067094 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:23.067069 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd" Oct 30 19:36:23.120531 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:23.120499 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:36:23.120531 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:23.120498 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:36:23.120955 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:23.120703 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:36:23.120955 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:23.120916 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:36:23.124950 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:23.124915 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd"] Oct 30 19:36:23.165047 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:23.165007 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlnsb\" (UniqueName: \"kubernetes.io/projected/32ae1753-20e4-4947-9bc5-5b52d6a2fbba-kube-api-access-nlnsb\") pod \"maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd\" (UID: \"32ae1753-20e4-4947-9bc5-5b52d6a2fbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd" Oct 30 19:36:23.165278 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:23.165062 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ae1753-20e4-4947-9bc5-5b52d6a2fbba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd\" (UID: \"32ae1753-20e4-4947-9bc5-5b52d6a2fbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd" Oct 30 19:36:23.165278 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:23.165242 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ae1753-20e4-4947-9bc5-5b52d6a2fbba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd\" (UID: \"32ae1753-20e4-4947-9bc5-5b52d6a2fbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd" Oct 30 19:36:23.266371 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:23.266323 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlnsb\" (UniqueName: \"kubernetes.io/projected/32ae1753-20e4-4947-9bc5-5b52d6a2fbba-kube-api-access-nlnsb\") pod \"maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd\" (UID: \"32ae1753-20e4-4947-9bc5-5b52d6a2fbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd" Oct 30 19:36:23.266582 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:23.266383 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ae1753-20e4-4947-9bc5-5b52d6a2fbba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd\" (UID: \"32ae1753-20e4-4947-9bc5-5b52d6a2fbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd" Oct 30 19:36:23.266582 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:23.266460 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ae1753-20e4-4947-9bc5-5b52d6a2fbba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd\" (UID: \"32ae1753-20e4-4947-9bc5-5b52d6a2fbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd" Oct 30 19:36:23.267088 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:23.267064 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ae1753-20e4-4947-9bc5-5b52d6a2fbba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd\" (UID: \"32ae1753-20e4-4947-9bc5-5b52d6a2fbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd" Oct 30 19:36:23.269222 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:23.269193 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ae1753-20e4-4947-9bc5-5b52d6a2fbba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd\" (UID: \"32ae1753-20e4-4947-9bc5-5b52d6a2fbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd" Oct 30 19:36:23.328289 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:23.328209 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlnsb\" (UniqueName: \"kubernetes.io/projected/32ae1753-20e4-4947-9bc5-5b52d6a2fbba-kube-api-access-nlnsb\") pod \"maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd\" (UID: \"32ae1753-20e4-4947-9bc5-5b52d6a2fbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd" Oct 30 19:36:23.381866 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:23.381827 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd" Oct 30 19:36:23.764449 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:23.764418 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd"] Oct 30 19:36:23.766459 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:36:23.766424 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ae1753_20e4_4947_9bc5_5b52d6a2fbba.slice/crio-c44741ec2c210a3b780d2f0eb79d7fd873a51c73035939f0a3f0a8ccee4ce615 WatchSource:0}: Error finding container c44741ec2c210a3b780d2f0eb79d7fd873a51c73035939f0a3f0a8ccee4ce615: Status 404 returned error can't find the container with id c44741ec2c210a3b780d2f0eb79d7fd873a51c73035939f0a3f0a8ccee4ce615 Oct 30 19:36:24.239013 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:24.238974 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd" event={"ID":"32ae1753-20e4-4947-9bc5-5b52d6a2fbba","Type":"ContainerStarted","Data":"14893e9b0337d55992ed002317bf01bcb5be32fd05ada85851645bd0c203b865"} Oct 30 19:36:24.239238 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:24.239037 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd" event={"ID":"32ae1753-20e4-4947-9bc5-5b52d6a2fbba","Type":"ContainerStarted","Data":"c44741ec2c210a3b780d2f0eb79d7fd873a51c73035939f0a3f0a8ccee4ce615"} Oct 30 19:36:24.368841 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:24.368724 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd" podStartSLOduration=1.368708372 podStartE2EDuration="1.368708372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:24.366296684 +0000 UTC m=+3744.393401643" watchObservedRunningTime="2026-10-30 19:36:24.368708372 +0000 UTC m=+3744.395813331" Oct 30 19:36:25.050426 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:25.050384 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7791f32-bghtn"] Oct 30 19:36:25.099421 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:25.099382 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7791f32-bghtn"] Oct 30 19:36:25.248014 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:25.247970 4531 generic.go:356] "Generic (PLEG): container finished" podID="32ae1753-20e4-4947-9bc5-5b52d6a2fbba" containerID="14893e9b0337d55992ed002317bf01bcb5be32fd05ada85851645bd0c203b865" exitCode=0 Oct 30 19:36:25.248258 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:25.248062 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd" event={"ID":"32ae1753-20e4-4947-9bc5-5b52d6a2fbba","Type":"ContainerDied","Data":"14893e9b0337d55992ed002317bf01bcb5be32fd05ada85851645bd0c203b865"} Oct 30 19:36:26.099821 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:26.099771 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e081c9c-cfd8-41bb-9aab-60e216e9ef1a" path="/var/lib/kubelet/pods/1e081c9c-cfd8-41bb-9aab-60e216e9ef1a/volumes" Oct 30 19:36:26.407511 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:36:26.407472 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:36:26.418660 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:26.418629 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd" Oct 30 19:36:26.503555 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:26.503514 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32ae1753-20e4-4947-9bc5-5b52d6a2fbba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ae1753-20e4-4947-9bc5-5b52d6a2fbba-maas-api-xks-ca\") pod \"32ae1753-20e4-4947-9bc5-5b52d6a2fbba\" (UID: \"32ae1753-20e4-4947-9bc5-5b52d6a2fbba\") " Oct 30 19:36:26.503762 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:26.503672 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32ae1753-20e4-4947-9bc5-5b52d6a2fbba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ae1753-20e4-4947-9bc5-5b52d6a2fbba-maas-api-openshift-ca\") pod \"32ae1753-20e4-4947-9bc5-5b52d6a2fbba\" (UID: \"32ae1753-20e4-4947-9bc5-5b52d6a2fbba\") " Oct 30 19:36:26.503762 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:26.503713 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ae1753-20e4-4947-9bc5-5b52d6a2fbba-kube-api-access-nlnsb\" (UniqueName: \"kubernetes.io/projected/32ae1753-20e4-4947-9bc5-5b52d6a2fbba-kube-api-access-nlnsb\") pod \"32ae1753-20e4-4947-9bc5-5b52d6a2fbba\" (UID: \"32ae1753-20e4-4947-9bc5-5b52d6a2fbba\") " Oct 30 19:36:26.504099 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:26.504069 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ae1753-20e4-4947-9bc5-5b52d6a2fbba-maas-api-openshift-ca" pod "32ae1753-20e4-4947-9bc5-5b52d6a2fbba" (UID: "32ae1753-20e4-4947-9bc5-5b52d6a2fbba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:26.506152 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:26.506112 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32ae1753-20e4-4947-9bc5-5b52d6a2fbba-maas-api-xks-ca" pod "32ae1753-20e4-4947-9bc5-5b52d6a2fbba" (UID: "32ae1753-20e4-4947-9bc5-5b52d6a2fbba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:36:26.506290 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:26.506211 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ae1753-20e4-4947-9bc5-5b52d6a2fbba-kube-api-access-nlnsb" pod "32ae1753-20e4-4947-9bc5-5b52d6a2fbba" (UID: "32ae1753-20e4-4947-9bc5-5b52d6a2fbba"). InnerVolumeSpecName "kube-api-access-nlnsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:26.605466 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:26.605410 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ae1753-20e4-4947-9bc5-5b52d6a2fbba-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:36:26.605466 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:26.605450 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlnsb\" (UniqueName: \"kubernetes.io/projected/32ae1753-20e4-4947-9bc5-5b52d6a2fbba-kube-api-access-nlnsb\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:36:26.605466 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:26.605474 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ae1753-20e4-4947-9bc5-5b52d6a2fbba-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:36:27.262877 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:27.262837 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd" Oct 30 19:36:27.263354 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:27.262902 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd" event={"ID":"32ae1753-20e4-4947-9bc5-5b52d6a2fbba","Type":"ContainerDied","Data":"c44741ec2c210a3b780d2f0eb79d7fd873a51c73035939f0a3f0a8ccee4ce615"} Oct 30 19:36:27.263354 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:27.262952 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c44741ec2c210a3b780d2f0eb79d7fd873a51c73035939f0a3f0a8ccee4ce615" Oct 30 19:36:27.296880 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:27.296823 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg" podUID="058ae453-9519-4149-80c9-8c8200d7b9f6" containerName="payload-processing" containerID="cri-o://0673de819d032a1f6a48f2f13cdfa1f0adf96f8e115d1e19edeca1a64b8bf73c" gracePeriod=25 Oct 30 19:36:27.406497 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:27.406437 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v" podUID="d666af3a-0ed2-4a2e-bed8-bde459fbdfd7" containerName="payload-pre-processing" containerID="cri-o://b5069e91edd9e97e86779599269cb6192a04378b72c74486b106d20b0969f374" gracePeriod=25 Oct 30 19:36:27.599422 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:27.599356 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg" Oct 30 19:36:27.699850 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:27.699812 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v" Oct 30 19:36:27.718454 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:27.718414 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/058ae453-9519-4149-80c9-8c8200d7b9f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058ae453-9519-4149-80c9-8c8200d7b9f6-plugins-config-volume\") pod \"058ae453-9519-4149-80c9-8c8200d7b9f6\" (UID: \"058ae453-9519-4149-80c9-8c8200d7b9f6\") " Oct 30 19:36:27.718602 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:27.718484 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/058ae453-9519-4149-80c9-8c8200d7b9f6-kube-api-access-tb7cb\" (UniqueName: \"kubernetes.io/projected/058ae453-9519-4149-80c9-8c8200d7b9f6-kube-api-access-tb7cb\") pod \"058ae453-9519-4149-80c9-8c8200d7b9f6\" (UID: \"058ae453-9519-4149-80c9-8c8200d7b9f6\") " Oct 30 19:36:27.718886 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:27.718859 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/058ae453-9519-4149-80c9-8c8200d7b9f6-plugins-config-volume" pod "058ae453-9519-4149-80c9-8c8200d7b9f6" (UID: "058ae453-9519-4149-80c9-8c8200d7b9f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:27.721234 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:27.721196 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/058ae453-9519-4149-80c9-8c8200d7b9f6-kube-api-access-tb7cb" pod "058ae453-9519-4149-80c9-8c8200d7b9f6" (UID: "058ae453-9519-4149-80c9-8c8200d7b9f6"). InnerVolumeSpecName "kube-api-access-tb7cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:27.820106 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:27.820007 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d666af3a-0ed2-4a2e-bed8-bde459fbdfd7-kube-api-access-szbx7\" (UniqueName: \"kubernetes.io/projected/d666af3a-0ed2-4a2e-bed8-bde459fbdfd7-kube-api-access-szbx7\") pod \"d666af3a-0ed2-4a2e-bed8-bde459fbdfd7\" (UID: \"d666af3a-0ed2-4a2e-bed8-bde459fbdfd7\") " Oct 30 19:36:27.820303 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:27.820113 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d666af3a-0ed2-4a2e-bed8-bde459fbdfd7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d666af3a-0ed2-4a2e-bed8-bde459fbdfd7-plugins-config-volume\") pod \"d666af3a-0ed2-4a2e-bed8-bde459fbdfd7\" (UID: \"d666af3a-0ed2-4a2e-bed8-bde459fbdfd7\") " Oct 30 19:36:27.820569 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:27.820544 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d666af3a-0ed2-4a2e-bed8-bde459fbdfd7-plugins-config-volume" pod "d666af3a-0ed2-4a2e-bed8-bde459fbdfd7" (UID: "d666af3a-0ed2-4a2e-bed8-bde459fbdfd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:27.820656 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:27.820638 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058ae453-9519-4149-80c9-8c8200d7b9f6-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:36:27.820697 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:27.820660 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d666af3a-0ed2-4a2e-bed8-bde459fbdfd7-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:36:27.820697 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:27.820672 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb7cb\" (UniqueName: \"kubernetes.io/projected/058ae453-9519-4149-80c9-8c8200d7b9f6-kube-api-access-tb7cb\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:36:27.822422 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:27.822374 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d666af3a-0ed2-4a2e-bed8-bde459fbdfd7-kube-api-access-szbx7" pod "d666af3a-0ed2-4a2e-bed8-bde459fbdfd7" (UID: "d666af3a-0ed2-4a2e-bed8-bde459fbdfd7"). InnerVolumeSpecName "kube-api-access-szbx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:27.921803 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:27.921760 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szbx7\" (UniqueName: \"kubernetes.io/projected/d666af3a-0ed2-4a2e-bed8-bde459fbdfd7-kube-api-access-szbx7\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:36:28.270982 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:28.270942 4531 generic.go:356] "Generic (PLEG): container finished" podID="d666af3a-0ed2-4a2e-bed8-bde459fbdfd7" containerID="b5069e91edd9e97e86779599269cb6192a04378b72c74486b106d20b0969f374" exitCode=0 Oct 30 19:36:28.271475 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:28.270994 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v" Oct 30 19:36:28.271475 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:28.271034 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v" event={"ID":"d666af3a-0ed2-4a2e-bed8-bde459fbdfd7","Type":"ContainerDied","Data":"b5069e91edd9e97e86779599269cb6192a04378b72c74486b106d20b0969f374"} Oct 30 19:36:28.271475 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:28.271092 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v" event={"ID":"d666af3a-0ed2-4a2e-bed8-bde459fbdfd7","Type":"ContainerDied","Data":"322daddfba962a4dacbd63f845e1b33b0ee7fb188167b035591f12c60a5f2db3"} Oct 30 19:36:28.271475 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:28.271124 4531 scope.go:122] "RemoveContainer" containerID="b5069e91edd9e97e86779599269cb6192a04378b72c74486b106d20b0969f374" Oct 30 19:36:28.272708 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:28.272673 4531 generic.go:356] "Generic (PLEG): container finished" podID="058ae453-9519-4149-80c9-8c8200d7b9f6" containerID="0673de819d032a1f6a48f2f13cdfa1f0adf96f8e115d1e19edeca1a64b8bf73c" exitCode=0 Oct 30 19:36:28.273004 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:28.272728 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg" Oct 30 19:36:28.273004 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:28.272763 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg" event={"ID":"058ae453-9519-4149-80c9-8c8200d7b9f6","Type":"ContainerDied","Data":"0673de819d032a1f6a48f2f13cdfa1f0adf96f8e115d1e19edeca1a64b8bf73c"} Oct 30 19:36:28.273004 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:28.272824 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg" event={"ID":"058ae453-9519-4149-80c9-8c8200d7b9f6","Type":"ContainerDied","Data":"459e46c4afd87aeb82f9693b78a693e93e94e1b01772b7d75c934030c6cc676b"} Oct 30 19:36:28.281889 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:28.281863 4531 scope.go:122] "RemoveContainer" containerID="b5069e91edd9e97e86779599269cb6192a04378b72c74486b106d20b0969f374" Oct 30 19:36:28.282293 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:36:28.282266 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5069e91edd9e97e86779599269cb6192a04378b72c74486b106d20b0969f374\": container with ID starting with b5069e91edd9e97e86779599269cb6192a04378b72c74486b106d20b0969f374 not found: ID does not exist" containerID="b5069e91edd9e97e86779599269cb6192a04378b72c74486b106d20b0969f374" Oct 30 19:36:28.282398 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:28.282300 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5069e91edd9e97e86779599269cb6192a04378b72c74486b106d20b0969f374"} err="failed to get container status \"b5069e91edd9e97e86779599269cb6192a04378b72c74486b106d20b0969f374\": rpc error: code = NotFound desc = could not find container \"b5069e91edd9e97e86779599269cb6192a04378b72c74486b106d20b0969f374\": container with ID starting with b5069e91edd9e97e86779599269cb6192a04378b72c74486b106d20b0969f374 not found: ID does not exist" Oct 30 19:36:28.282398 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:28.282324 4531 scope.go:122] "RemoveContainer" containerID="0673de819d032a1f6a48f2f13cdfa1f0adf96f8e115d1e19edeca1a64b8bf73c" Oct 30 19:36:28.292755 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:28.292726 4531 scope.go:122] "RemoveContainer" containerID="0673de819d032a1f6a48f2f13cdfa1f0adf96f8e115d1e19edeca1a64b8bf73c" Oct 30 19:36:28.293188 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:36:28.293163 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0673de819d032a1f6a48f2f13cdfa1f0adf96f8e115d1e19edeca1a64b8bf73c\": container with ID starting with 0673de819d032a1f6a48f2f13cdfa1f0adf96f8e115d1e19edeca1a64b8bf73c not found: ID does not exist" containerID="0673de819d032a1f6a48f2f13cdfa1f0adf96f8e115d1e19edeca1a64b8bf73c" Oct 30 19:36:28.293273 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:28.293200 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0673de819d032a1f6a48f2f13cdfa1f0adf96f8e115d1e19edeca1a64b8bf73c"} err="failed to get container status \"0673de819d032a1f6a48f2f13cdfa1f0adf96f8e115d1e19edeca1a64b8bf73c\": rpc error: code = NotFound desc = could not find container \"0673de819d032a1f6a48f2f13cdfa1f0adf96f8e115d1e19edeca1a64b8bf73c\": container with ID starting with 0673de819d032a1f6a48f2f13cdfa1f0adf96f8e115d1e19edeca1a64b8bf73c not found: ID does not exist" Oct 30 19:36:28.441163 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:28.441109 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg"] Oct 30 19:36:28.486790 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:28.486748 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-da1b8b-86bfdf4b48-q7svg"] Oct 30 19:36:28.556826 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:28.556788 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v"] Oct 30 19:36:28.561257 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:28.561219 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-da1b8b-57cbcb89bc-86d4v"] Oct 30 19:36:30.100715 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:30.100664 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="058ae453-9519-4149-80c9-8c8200d7b9f6" path="/var/lib/kubelet/pods/058ae453-9519-4149-80c9-8c8200d7b9f6/volumes" Oct 30 19:36:30.101309 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:30.101286 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d666af3a-0ed2-4a2e-bed8-bde459fbdfd7" path="/var/lib/kubelet/pods/d666af3a-0ed2-4a2e-bed8-bde459fbdfd7/volumes" Oct 30 19:36:37.148618 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:37.148577 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg"] Oct 30 19:36:42.149890 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:42.149800 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg" podUID="16cc1cfb-007c-4332-aad6-1d0b22eb71b8" containerName="payload-pre-processing" containerID="cri-o://34689f13d7c9eb15b4da3d1ca34c94092b0cad84f476690b7c7208ddea680fed" gracePeriod=25 Oct 30 19:36:42.371310 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:42.371276 4531 generic.go:356] "Generic (PLEG): container finished" podID="16cc1cfb-007c-4332-aad6-1d0b22eb71b8" containerID="34689f13d7c9eb15b4da3d1ca34c94092b0cad84f476690b7c7208ddea680fed" exitCode=0 Oct 30 19:36:42.371529 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:42.371358 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg" event={"ID":"16cc1cfb-007c-4332-aad6-1d0b22eb71b8","Type":"ContainerDied","Data":"34689f13d7c9eb15b4da3d1ca34c94092b0cad84f476690b7c7208ddea680fed"} Oct 30 19:36:42.416089 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:42.416013 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg" Oct 30 19:36:42.594387 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:42.594345 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16cc1cfb-007c-4332-aad6-1d0b22eb71b8-kube-api-access-hhz58\" (UniqueName: \"kubernetes.io/projected/16cc1cfb-007c-4332-aad6-1d0b22eb71b8-kube-api-access-hhz58\") pod \"16cc1cfb-007c-4332-aad6-1d0b22eb71b8\" (UID: \"16cc1cfb-007c-4332-aad6-1d0b22eb71b8\") " Oct 30 19:36:42.594594 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:42.594450 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16cc1cfb-007c-4332-aad6-1d0b22eb71b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16cc1cfb-007c-4332-aad6-1d0b22eb71b8-plugins-config-volume\") pod \"16cc1cfb-007c-4332-aad6-1d0b22eb71b8\" (UID: \"16cc1cfb-007c-4332-aad6-1d0b22eb71b8\") " Oct 30 19:36:42.594904 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:42.594877 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16cc1cfb-007c-4332-aad6-1d0b22eb71b8-plugins-config-volume" pod "16cc1cfb-007c-4332-aad6-1d0b22eb71b8" (UID: "16cc1cfb-007c-4332-aad6-1d0b22eb71b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:42.596725 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:42.596705 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16cc1cfb-007c-4332-aad6-1d0b22eb71b8-kube-api-access-hhz58" pod "16cc1cfb-007c-4332-aad6-1d0b22eb71b8" (UID: "16cc1cfb-007c-4332-aad6-1d0b22eb71b8"). InnerVolumeSpecName "kube-api-access-hhz58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:42.695670 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:42.695574 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhz58\" (UniqueName: \"kubernetes.io/projected/16cc1cfb-007c-4332-aad6-1d0b22eb71b8-kube-api-access-hhz58\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:36:42.695670 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:42.695604 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16cc1cfb-007c-4332-aad6-1d0b22eb71b8-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:36:43.379477 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:43.379446 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg" Oct 30 19:36:43.379974 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:43.379497 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg" event={"ID":"16cc1cfb-007c-4332-aad6-1d0b22eb71b8","Type":"ContainerDied","Data":"4a00fecdd7c3fabb12407c2ac00f6f62fcb8a9670d9eea9ff74c377c0de78eb9"} Oct 30 19:36:43.379974 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:43.379560 4531 scope.go:122] "RemoveContainer" containerID="34689f13d7c9eb15b4da3d1ca34c94092b0cad84f476690b7c7208ddea680fed" Oct 30 19:36:43.544590 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:43.544550 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg"] Oct 30 19:36:43.595423 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:43.595385 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-7fdf6b-76c6d69c99-jg8zg"] Oct 30 19:36:44.099641 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:36:44.099604 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16cc1cfb-007c-4332-aad6-1d0b22eb71b8" path="/var/lib/kubelet/pods/16cc1cfb-007c-4332-aad6-1d0b22eb71b8/volumes" Oct 30 19:36:56.399678 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:36:56.399642 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:37:03.065319 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:03.065283 4531 scope.go:122] "RemoveContainer" containerID="d6f4639d2ec4cf8bb01a1609e00040eebb92b92ef6c12d1d9831605b3e5efcdb" Oct 30 19:37:03.075757 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:03.075734 4531 scope.go:122] "RemoveContainer" containerID="50926cb9eec74d69bfc26aa3057f228c30dee2e2f1408151606ec38be628fc1b" Oct 30 19:37:26.398928 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:37:26.398885 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:37:28.716969 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.716926 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h"] Oct 30 19:37:28.717549 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.717530 4531 cpu_manager.go:414] "containerMap: removing container" podUID="d666af3a-0ed2-4a2e-bed8-bde459fbdfd7" containerName="payload-pre-processing" Oct 30 19:37:28.717601 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.717551 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d666af3a-0ed2-4a2e-bed8-bde459fbdfd7" containerName="payload-pre-processing" Oct 30 19:37:28.717601 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.717566 4531 cpu_manager.go:414] "containerMap: removing container" podUID="16cc1cfb-007c-4332-aad6-1d0b22eb71b8" containerName="payload-pre-processing" Oct 30 19:37:28.717601 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.717571 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16cc1cfb-007c-4332-aad6-1d0b22eb71b8" containerName="payload-pre-processing" Oct 30 19:37:28.717601 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.717586 4531 cpu_manager.go:414] "containerMap: removing container" podUID="058ae453-9519-4149-80c9-8c8200d7b9f6" containerName="payload-processing" Oct 30 19:37:28.717601 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.717592 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="058ae453-9519-4149-80c9-8c8200d7b9f6" containerName="payload-processing" Oct 30 19:37:28.717776 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.717621 4531 cpu_manager.go:414] "containerMap: removing container" podUID="32ae1753-20e4-4947-9bc5-5b52d6a2fbba" containerName="revoke-keys" Oct 30 19:37:28.717776 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.717626 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ae1753-20e4-4947-9bc5-5b52d6a2fbba" containerName="revoke-keys" Oct 30 19:37:28.717776 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.717709 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d666af3a-0ed2-4a2e-bed8-bde459fbdfd7" containerName="payload-pre-processing" Oct 30 19:37:28.717776 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.717718 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="16cc1cfb-007c-4332-aad6-1d0b22eb71b8" containerName="payload-pre-processing" Oct 30 19:37:28.717776 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.717725 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="058ae453-9519-4149-80c9-8c8200d7b9f6" containerName="payload-processing" Oct 30 19:37:28.717776 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.717734 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="32ae1753-20e4-4947-9bc5-5b52d6a2fbba" containerName="revoke-keys" Oct 30 19:37:28.728333 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.728296 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h" Oct 30 19:37:28.765627 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.765592 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h"] Oct 30 19:37:28.782036 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.781985 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:37:28.782036 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.782017 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:37:28.782284 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.782112 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:37:28.811851 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.811811 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:37:28.817390 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.817361 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a9ce5a7-266b-4e63-8fd5-1630a0bd0931-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h\" (UID: \"5a9ce5a7-266b-4e63-8fd5-1630a0bd0931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h" Oct 30 19:37:28.817519 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.817419 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6zvc\" (UniqueName: \"kubernetes.io/projected/5a9ce5a7-266b-4e63-8fd5-1630a0bd0931-kube-api-access-k6zvc\") pod \"maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h\" (UID: \"5a9ce5a7-266b-4e63-8fd5-1630a0bd0931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h" Oct 30 19:37:28.817519 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.817508 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a9ce5a7-266b-4e63-8fd5-1630a0bd0931-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h\" (UID: \"5a9ce5a7-266b-4e63-8fd5-1630a0bd0931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h" Oct 30 19:37:28.918732 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.918691 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a9ce5a7-266b-4e63-8fd5-1630a0bd0931-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h\" (UID: \"5a9ce5a7-266b-4e63-8fd5-1630a0bd0931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h" Oct 30 19:37:28.918948 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.918763 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6zvc\" (UniqueName: \"kubernetes.io/projected/5a9ce5a7-266b-4e63-8fd5-1630a0bd0931-kube-api-access-k6zvc\") pod \"maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h\" (UID: \"5a9ce5a7-266b-4e63-8fd5-1630a0bd0931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h" Oct 30 19:37:28.918948 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.918787 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a9ce5a7-266b-4e63-8fd5-1630a0bd0931-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h\" (UID: \"5a9ce5a7-266b-4e63-8fd5-1630a0bd0931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h" Oct 30 19:37:28.919490 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.919465 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a9ce5a7-266b-4e63-8fd5-1630a0bd0931-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h\" (UID: \"5a9ce5a7-266b-4e63-8fd5-1630a0bd0931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h" Oct 30 19:37:28.921340 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.921316 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a9ce5a7-266b-4e63-8fd5-1630a0bd0931-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h\" (UID: \"5a9ce5a7-266b-4e63-8fd5-1630a0bd0931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h" Oct 30 19:37:28.980633 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:28.980553 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6zvc\" (UniqueName: \"kubernetes.io/projected/5a9ce5a7-266b-4e63-8fd5-1630a0bd0931-kube-api-access-k6zvc\") pod \"maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h\" (UID: \"5a9ce5a7-266b-4e63-8fd5-1630a0bd0931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h" Oct 30 19:37:29.042064 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:29.042027 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h" Oct 30 19:37:29.418476 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:29.418447 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h"] Oct 30 19:37:29.421008 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:37:29.420976 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a9ce5a7_266b_4e63_8fd5_1630a0bd0931.slice/crio-bd04ca625cbaf4d14234de1652026e51ef933c48e69b4044f932565bf93c178a WatchSource:0}: Error finding container bd04ca625cbaf4d14234de1652026e51ef933c48e69b4044f932565bf93c178a: Status 404 returned error can't find the container with id bd04ca625cbaf4d14234de1652026e51ef933c48e69b4044f932565bf93c178a Oct 30 19:37:29.677284 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:29.677184 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h" event={"ID":"5a9ce5a7-266b-4e63-8fd5-1630a0bd0931","Type":"ContainerStarted","Data":"bd04ca625cbaf4d14234de1652026e51ef933c48e69b4044f932565bf93c178a"} Oct 30 19:37:30.000238 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:30.000206 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs"] Oct 30 19:37:30.049704 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:30.049672 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-da1b8b-77zhs"] Oct 30 19:37:30.098923 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:30.098891 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a078ee1-bc3f-4e8f-8a10-e319d18c3030" path="/var/lib/kubelet/pods/7a078ee1-bc3f-4e8f-8a10-e319d18c3030/volumes" Oct 30 19:37:30.685849 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:30.685757 4531 generic.go:356] "Generic (PLEG): container finished" podID="5a9ce5a7-266b-4e63-8fd5-1630a0bd0931" containerID="0a61a0ca5c04d6050791ccd17ed7b264efff77c105cc15dbfbc2794bbc9d93e3" exitCode=0 Oct 30 19:37:30.685849 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:30.685826 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h" event={"ID":"5a9ce5a7-266b-4e63-8fd5-1630a0bd0931","Type":"ContainerDied","Data":"0a61a0ca5c04d6050791ccd17ed7b264efff77c105cc15dbfbc2794bbc9d93e3"} Oct 30 19:37:31.837419 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:31.837383 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h" Oct 30 19:37:31.953432 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:31.953336 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a9ce5a7-266b-4e63-8fd5-1630a0bd0931-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a9ce5a7-266b-4e63-8fd5-1630a0bd0931-maas-api-openshift-ca\") pod \"5a9ce5a7-266b-4e63-8fd5-1630a0bd0931\" (UID: \"5a9ce5a7-266b-4e63-8fd5-1630a0bd0931\") " Oct 30 19:37:31.953432 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:31.953406 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a9ce5a7-266b-4e63-8fd5-1630a0bd0931-kube-api-access-k6zvc\" (UniqueName: \"kubernetes.io/projected/5a9ce5a7-266b-4e63-8fd5-1630a0bd0931-kube-api-access-k6zvc\") pod \"5a9ce5a7-266b-4e63-8fd5-1630a0bd0931\" (UID: \"5a9ce5a7-266b-4e63-8fd5-1630a0bd0931\") " Oct 30 19:37:31.953661 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:31.953443 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a9ce5a7-266b-4e63-8fd5-1630a0bd0931-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a9ce5a7-266b-4e63-8fd5-1630a0bd0931-maas-api-xks-ca\") pod \"5a9ce5a7-266b-4e63-8fd5-1630a0bd0931\" (UID: \"5a9ce5a7-266b-4e63-8fd5-1630a0bd0931\") " Oct 30 19:37:31.953802 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:31.953777 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a9ce5a7-266b-4e63-8fd5-1630a0bd0931-maas-api-openshift-ca" pod "5a9ce5a7-266b-4e63-8fd5-1630a0bd0931" (UID: "5a9ce5a7-266b-4e63-8fd5-1630a0bd0931"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:31.955828 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:31.955797 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a9ce5a7-266b-4e63-8fd5-1630a0bd0931-maas-api-xks-ca" pod "5a9ce5a7-266b-4e63-8fd5-1630a0bd0931" (UID: "5a9ce5a7-266b-4e63-8fd5-1630a0bd0931"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:37:31.955981 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:31.955910 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9ce5a7-266b-4e63-8fd5-1630a0bd0931-kube-api-access-k6zvc" pod "5a9ce5a7-266b-4e63-8fd5-1630a0bd0931" (UID: "5a9ce5a7-266b-4e63-8fd5-1630a0bd0931"). InnerVolumeSpecName "kube-api-access-k6zvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:32.054746 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:32.054706 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a9ce5a7-266b-4e63-8fd5-1630a0bd0931-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:32.054746 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:32.054743 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a9ce5a7-266b-4e63-8fd5-1630a0bd0931-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:32.054977 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:32.054755 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6zvc\" (UniqueName: \"kubernetes.io/projected/5a9ce5a7-266b-4e63-8fd5-1630a0bd0931-kube-api-access-k6zvc\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:32.699274 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:32.699231 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h" event={"ID":"5a9ce5a7-266b-4e63-8fd5-1630a0bd0931","Type":"ContainerDied","Data":"bd04ca625cbaf4d14234de1652026e51ef933c48e69b4044f932565bf93c178a"} Oct 30 19:37:32.699465 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:32.699315 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd04ca625cbaf4d14234de1652026e51ef933c48e69b4044f932565bf93c178a" Oct 30 19:37:32.699465 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:32.699252 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h" Oct 30 19:37:33.672255 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:33.672219 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm"] Oct 30 19:37:33.672955 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:33.672932 4531 cpu_manager.go:414] "containerMap: removing container" podUID="5a9ce5a7-266b-4e63-8fd5-1630a0bd0931" containerName="revoke-keys" Oct 30 19:37:33.673063 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:33.672958 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a9ce5a7-266b-4e63-8fd5-1630a0bd0931" containerName="revoke-keys" Oct 30 19:37:33.673164 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:33.673142 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a9ce5a7-266b-4e63-8fd5-1630a0bd0931" containerName="revoke-keys" Oct 30 19:37:33.676408 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:33.676388 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm" Oct 30 19:37:33.733881 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:33.733848 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:37:33.737729 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:33.737698 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm"] Oct 30 19:37:33.774698 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:33.774654 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c0dff6-715a-4d21-a4a6-185976b86774-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm\" (UID: \"d5c0dff6-715a-4d21-a4a6-185976b86774\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm" Oct 30 19:37:33.774884 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:33.774712 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c0dff6-715a-4d21-a4a6-185976b86774-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm\" (UID: \"d5c0dff6-715a-4d21-a4a6-185976b86774\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm" Oct 30 19:37:33.774884 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:33.774770 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rfvz\" (UniqueName: \"kubernetes.io/projected/d5c0dff6-715a-4d21-a4a6-185976b86774-kube-api-access-2rfvz\") pod \"maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm\" (UID: \"d5c0dff6-715a-4d21-a4a6-185976b86774\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm" Oct 30 19:37:33.782633 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:33.782611 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:37:33.782723 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:33.782611 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:37:33.783352 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:33.783331 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:37:33.876228 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:33.876188 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c0dff6-715a-4d21-a4a6-185976b86774-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm\" (UID: \"d5c0dff6-715a-4d21-a4a6-185976b86774\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm" Oct 30 19:37:33.876228 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:33.876230 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rfvz\" (UniqueName: \"kubernetes.io/projected/d5c0dff6-715a-4d21-a4a6-185976b86774-kube-api-access-2rfvz\") pod \"maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm\" (UID: \"d5c0dff6-715a-4d21-a4a6-185976b86774\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm" Oct 30 19:37:33.876462 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:33.876353 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c0dff6-715a-4d21-a4a6-185976b86774-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm\" (UID: \"d5c0dff6-715a-4d21-a4a6-185976b86774\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm" Oct 30 19:37:33.876998 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:33.876955 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c0dff6-715a-4d21-a4a6-185976b86774-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm\" (UID: \"d5c0dff6-715a-4d21-a4a6-185976b86774\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm" Oct 30 19:37:33.878803 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:33.878761 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c0dff6-715a-4d21-a4a6-185976b86774-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm\" (UID: \"d5c0dff6-715a-4d21-a4a6-185976b86774\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm" Oct 30 19:37:33.939405 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:33.939328 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rfvz\" (UniqueName: \"kubernetes.io/projected/d5c0dff6-715a-4d21-a4a6-185976b86774-kube-api-access-2rfvz\") pod \"maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm\" (UID: \"d5c0dff6-715a-4d21-a4a6-185976b86774\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm" Oct 30 19:37:33.991341 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:33.991299 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm" Oct 30 19:37:34.380744 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:34.380710 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm"] Oct 30 19:37:34.382988 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:37:34.382944 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c0dff6_715a_4d21_a4a6_185976b86774.slice/crio-73ef506fd570c9641b8cdc56b8450462bcb152f942708e07d5dcccaeb3c746a4 WatchSource:0}: Error finding container 73ef506fd570c9641b8cdc56b8450462bcb152f942708e07d5dcccaeb3c746a4: Status 404 returned error can't find the container with id 73ef506fd570c9641b8cdc56b8450462bcb152f942708e07d5dcccaeb3c746a4 Oct 30 19:37:34.714945 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:34.714850 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm" event={"ID":"d5c0dff6-715a-4d21-a4a6-185976b86774","Type":"ContainerStarted","Data":"73ef506fd570c9641b8cdc56b8450462bcb152f942708e07d5dcccaeb3c746a4"} Oct 30 19:37:35.723681 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:35.723577 4531 generic.go:356] "Generic (PLEG): container finished" podID="d5c0dff6-715a-4d21-a4a6-185976b86774" containerID="fa6d67075ebe66e51529ef6f3f2f770dc12fc8cd42aa15f57882a9befa52007e" exitCode=0 Oct 30 19:37:35.723681 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:35.723654 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm" event={"ID":"d5c0dff6-715a-4d21-a4a6-185976b86774","Type":"ContainerDied","Data":"fa6d67075ebe66e51529ef6f3f2f770dc12fc8cd42aa15f57882a9befa52007e"} Oct 30 19:37:36.884627 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:36.884594 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm" Oct 30 19:37:37.014990 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:37.014954 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5c0dff6-715a-4d21-a4a6-185976b86774-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c0dff6-715a-4d21-a4a6-185976b86774-maas-api-xks-ca\") pod \"d5c0dff6-715a-4d21-a4a6-185976b86774\" (UID: \"d5c0dff6-715a-4d21-a4a6-185976b86774\") " Oct 30 19:37:37.015231 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:37.015068 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5c0dff6-715a-4d21-a4a6-185976b86774-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c0dff6-715a-4d21-a4a6-185976b86774-maas-api-openshift-ca\") pod \"d5c0dff6-715a-4d21-a4a6-185976b86774\" (UID: \"d5c0dff6-715a-4d21-a4a6-185976b86774\") " Oct 30 19:37:37.015231 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:37.015098 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c0dff6-715a-4d21-a4a6-185976b86774-kube-api-access-2rfvz\" (UniqueName: \"kubernetes.io/projected/d5c0dff6-715a-4d21-a4a6-185976b86774-kube-api-access-2rfvz\") pod \"d5c0dff6-715a-4d21-a4a6-185976b86774\" (UID: \"d5c0dff6-715a-4d21-a4a6-185976b86774\") " Oct 30 19:37:37.015628 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:37.015590 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c0dff6-715a-4d21-a4a6-185976b86774-maas-api-openshift-ca" pod "d5c0dff6-715a-4d21-a4a6-185976b86774" (UID: "d5c0dff6-715a-4d21-a4a6-185976b86774"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:37.017640 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:37.017603 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c0dff6-715a-4d21-a4a6-185976b86774-maas-api-xks-ca" pod "d5c0dff6-715a-4d21-a4a6-185976b86774" (UID: "d5c0dff6-715a-4d21-a4a6-185976b86774"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:37:37.017772 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:37.017648 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c0dff6-715a-4d21-a4a6-185976b86774-kube-api-access-2rfvz" pod "d5c0dff6-715a-4d21-a4a6-185976b86774" (UID: "d5c0dff6-715a-4d21-a4a6-185976b86774"). InnerVolumeSpecName "kube-api-access-2rfvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:37.116280 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:37.116229 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c0dff6-715a-4d21-a4a6-185976b86774-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:37.116280 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:37.116266 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rfvz\" (UniqueName: \"kubernetes.io/projected/d5c0dff6-715a-4d21-a4a6-185976b86774-kube-api-access-2rfvz\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:37.116280 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:37.116282 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c0dff6-715a-4d21-a4a6-185976b86774-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:37.738695 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:37.738652 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm" event={"ID":"d5c0dff6-715a-4d21-a4a6-185976b86774","Type":"ContainerDied","Data":"73ef506fd570c9641b8cdc56b8450462bcb152f942708e07d5dcccaeb3c746a4"} Oct 30 19:37:37.739033 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:37.739008 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73ef506fd570c9641b8cdc56b8450462bcb152f942708e07d5dcccaeb3c746a4" Oct 30 19:37:37.739033 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:37.738675 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm" Oct 30 19:37:38.715768 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:38.715728 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b"] Oct 30 19:37:38.716328 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:38.716310 4531 cpu_manager.go:414] "containerMap: removing container" podUID="d5c0dff6-715a-4d21-a4a6-185976b86774" containerName="revoke-keys" Oct 30 19:37:38.716398 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:38.716330 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c0dff6-715a-4d21-a4a6-185976b86774" containerName="revoke-keys" Oct 30 19:37:38.716453 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:38.716439 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5c0dff6-715a-4d21-a4a6-185976b86774" containerName="revoke-keys" Oct 30 19:37:38.719832 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:38.719811 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b" Oct 30 19:37:38.778491 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:38.778451 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:37:38.828238 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:38.828199 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:37:38.828498 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:38.828481 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:37:38.828580 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:38.828569 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:37:38.831001 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:38.830968 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b"] Oct 30 19:37:38.836283 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:38.836248 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c428fcd4-f859-416e-913c-4f719d7ded51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b\" (UID: \"c428fcd4-f859-416e-913c-4f719d7ded51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b" Oct 30 19:37:38.836379 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:38.836313 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qmjr\" (UniqueName: \"kubernetes.io/projected/c428fcd4-f859-416e-913c-4f719d7ded51-kube-api-access-5qmjr\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b\" (UID: \"c428fcd4-f859-416e-913c-4f719d7ded51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b" Oct 30 19:37:38.836527 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:38.836507 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c428fcd4-f859-416e-913c-4f719d7ded51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b\" (UID: \"c428fcd4-f859-416e-913c-4f719d7ded51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b" Oct 30 19:37:38.937957 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:38.937912 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c428fcd4-f859-416e-913c-4f719d7ded51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b\" (UID: \"c428fcd4-f859-416e-913c-4f719d7ded51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b" Oct 30 19:37:38.938220 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:38.938008 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c428fcd4-f859-416e-913c-4f719d7ded51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b\" (UID: \"c428fcd4-f859-416e-913c-4f719d7ded51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b" Oct 30 19:37:38.938220 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:38.938065 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qmjr\" (UniqueName: \"kubernetes.io/projected/c428fcd4-f859-416e-913c-4f719d7ded51-kube-api-access-5qmjr\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b\" (UID: \"c428fcd4-f859-416e-913c-4f719d7ded51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b" Oct 30 19:37:38.938812 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:38.938780 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c428fcd4-f859-416e-913c-4f719d7ded51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b\" (UID: \"c428fcd4-f859-416e-913c-4f719d7ded51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b" Oct 30 19:37:38.940777 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:38.940754 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c428fcd4-f859-416e-913c-4f719d7ded51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b\" (UID: \"c428fcd4-f859-416e-913c-4f719d7ded51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b" Oct 30 19:37:38.999253 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:38.999212 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qmjr\" (UniqueName: \"kubernetes.io/projected/c428fcd4-f859-416e-913c-4f719d7ded51-kube-api-access-5qmjr\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b\" (UID: \"c428fcd4-f859-416e-913c-4f719d7ded51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b" Oct 30 19:37:39.034298 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:39.034257 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b" Oct 30 19:37:39.415959 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:39.415933 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b"] Oct 30 19:37:39.417351 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:37:39.417322 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc428fcd4_f859_416e_913c_4f719d7ded51.slice/crio-77ab769d23c28928801c7eecfcb012eb955db50e67c73c5e1216598106ca100b WatchSource:0}: Error finding container 77ab769d23c28928801c7eecfcb012eb955db50e67c73c5e1216598106ca100b: Status 404 returned error can't find the container with id 77ab769d23c28928801c7eecfcb012eb955db50e67c73c5e1216598106ca100b Oct 30 19:37:39.754465 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:39.754413 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b" event={"ID":"c428fcd4-f859-416e-913c-4f719d7ded51","Type":"ContainerStarted","Data":"77ab769d23c28928801c7eecfcb012eb955db50e67c73c5e1216598106ca100b"} Oct 30 19:37:39.849415 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:39.849357 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m"] Oct 30 19:37:39.959479 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:39.959434 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs"] Oct 30 19:37:40.761950 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:40.761915 4531 generic.go:356] "Generic (PLEG): container finished" podID="c428fcd4-f859-416e-913c-4f719d7ded51" containerID="d37c795f487c10a5071def6abb86995dc2cb3635eb721f7fe4f5094dcff62ab4" exitCode=0 Oct 30 19:37:40.762410 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:40.762000 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b" event={"ID":"c428fcd4-f859-416e-913c-4f719d7ded51","Type":"ContainerDied","Data":"d37c795f487c10a5071def6abb86995dc2cb3635eb721f7fe4f5094dcff62ab4"} Oct 30 19:37:41.940520 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:41.940487 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b" Oct 30 19:37:42.072501 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:42.072417 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c428fcd4-f859-416e-913c-4f719d7ded51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c428fcd4-f859-416e-913c-4f719d7ded51-maas-api-openshift-ca\") pod \"c428fcd4-f859-416e-913c-4f719d7ded51\" (UID: \"c428fcd4-f859-416e-913c-4f719d7ded51\") " Oct 30 19:37:42.072501 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:42.072457 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c428fcd4-f859-416e-913c-4f719d7ded51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c428fcd4-f859-416e-913c-4f719d7ded51-maas-api-xks-ca\") pod \"c428fcd4-f859-416e-913c-4f719d7ded51\" (UID: \"c428fcd4-f859-416e-913c-4f719d7ded51\") " Oct 30 19:37:42.072731 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:42.072554 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c428fcd4-f859-416e-913c-4f719d7ded51-kube-api-access-5qmjr\" (UniqueName: \"kubernetes.io/projected/c428fcd4-f859-416e-913c-4f719d7ded51-kube-api-access-5qmjr\") pod \"c428fcd4-f859-416e-913c-4f719d7ded51\" (UID: \"c428fcd4-f859-416e-913c-4f719d7ded51\") " Oct 30 19:37:42.072914 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:42.072883 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c428fcd4-f859-416e-913c-4f719d7ded51-maas-api-openshift-ca" pod "c428fcd4-f859-416e-913c-4f719d7ded51" (UID: "c428fcd4-f859-416e-913c-4f719d7ded51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:42.075100 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:42.075061 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c428fcd4-f859-416e-913c-4f719d7ded51-maas-api-xks-ca" pod "c428fcd4-f859-416e-913c-4f719d7ded51" (UID: "c428fcd4-f859-416e-913c-4f719d7ded51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:37:42.075315 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:42.075246 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c428fcd4-f859-416e-913c-4f719d7ded51-kube-api-access-5qmjr" pod "c428fcd4-f859-416e-913c-4f719d7ded51" (UID: "c428fcd4-f859-416e-913c-4f719d7ded51"). InnerVolumeSpecName "kube-api-access-5qmjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:42.174381 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:42.174346 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c428fcd4-f859-416e-913c-4f719d7ded51-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:42.174381 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:42.174378 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c428fcd4-f859-416e-913c-4f719d7ded51-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:42.174381 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:42.174391 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qmjr\" (UniqueName: \"kubernetes.io/projected/c428fcd4-f859-416e-913c-4f719d7ded51-kube-api-access-5qmjr\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:42.776299 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:42.776266 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b" Oct 30 19:37:42.776299 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:42.776287 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b" event={"ID":"c428fcd4-f859-416e-913c-4f719d7ded51","Type":"ContainerDied","Data":"77ab769d23c28928801c7eecfcb012eb955db50e67c73c5e1216598106ca100b"} Oct 30 19:37:42.776539 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:42.776344 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77ab769d23c28928801c7eecfcb012eb955db50e67c73c5e1216598106ca100b" Oct 30 19:37:44.350293 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:44.350253 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7"] Oct 30 19:37:44.462290 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:44.462244 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj"] Oct 30 19:37:44.850710 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:44.850649 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m" podUID="387d077d-6792-45c0-ae7c-2891c00f47c2" containerName="payload-processing" containerID="cri-o://9186cfce414a0a851a046cf9da2260bfa8a94b50215af55b550cf564ad729866" gracePeriod=25 Oct 30 19:37:44.960120 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:44.960068 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs" podUID="d9f38975-1b9b-47d0-93c1-0ab25c13d127" containerName="payload-pre-processing" containerID="cri-o://4db96304fc3ebc271636b13adcdcd46301d9642c20755393c8c89098c14a4f97" gracePeriod=25 Oct 30 19:37:45.156659 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.156304 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m" Oct 30 19:37:45.209075 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.209041 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/387d077d-6792-45c0-ae7c-2891c00f47c2-kube-api-access-mxxn7\" (UniqueName: \"kubernetes.io/projected/387d077d-6792-45c0-ae7c-2891c00f47c2-kube-api-access-mxxn7\") pod \"387d077d-6792-45c0-ae7c-2891c00f47c2\" (UID: \"387d077d-6792-45c0-ae7c-2891c00f47c2\") " Oct 30 19:37:45.209308 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.209118 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/387d077d-6792-45c0-ae7c-2891c00f47c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387d077d-6792-45c0-ae7c-2891c00f47c2-plugins-config-volume\") pod \"387d077d-6792-45c0-ae7c-2891c00f47c2\" (UID: \"387d077d-6792-45c0-ae7c-2891c00f47c2\") " Oct 30 19:37:45.209752 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.209716 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/387d077d-6792-45c0-ae7c-2891c00f47c2-plugins-config-volume" pod "387d077d-6792-45c0-ae7c-2891c00f47c2" (UID: "387d077d-6792-45c0-ae7c-2891c00f47c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:45.213360 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.213327 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/387d077d-6792-45c0-ae7c-2891c00f47c2-kube-api-access-mxxn7" pod "387d077d-6792-45c0-ae7c-2891c00f47c2" (UID: "387d077d-6792-45c0-ae7c-2891c00f47c2"). InnerVolumeSpecName "kube-api-access-mxxn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:45.253107 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.253077 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs" Oct 30 19:37:45.309955 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.309917 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f38975-1b9b-47d0-93c1-0ab25c13d127-kube-api-access-bvhmt\" (UniqueName: \"kubernetes.io/projected/d9f38975-1b9b-47d0-93c1-0ab25c13d127-kube-api-access-bvhmt\") pod \"d9f38975-1b9b-47d0-93c1-0ab25c13d127\" (UID: \"d9f38975-1b9b-47d0-93c1-0ab25c13d127\") " Oct 30 19:37:45.310211 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.310020 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9f38975-1b9b-47d0-93c1-0ab25c13d127-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f38975-1b9b-47d0-93c1-0ab25c13d127-plugins-config-volume\") pod \"d9f38975-1b9b-47d0-93c1-0ab25c13d127\" (UID: \"d9f38975-1b9b-47d0-93c1-0ab25c13d127\") " Oct 30 19:37:45.310342 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.310325 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxxn7\" (UniqueName: \"kubernetes.io/projected/387d077d-6792-45c0-ae7c-2891c00f47c2-kube-api-access-mxxn7\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:45.310485 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.310343 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387d077d-6792-45c0-ae7c-2891c00f47c2-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:45.310485 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.310429 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f38975-1b9b-47d0-93c1-0ab25c13d127-plugins-config-volume" pod "d9f38975-1b9b-47d0-93c1-0ab25c13d127" (UID: "d9f38975-1b9b-47d0-93c1-0ab25c13d127"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:45.312369 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.312337 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f38975-1b9b-47d0-93c1-0ab25c13d127-kube-api-access-bvhmt" pod "d9f38975-1b9b-47d0-93c1-0ab25c13d127" (UID: "d9f38975-1b9b-47d0-93c1-0ab25c13d127"). InnerVolumeSpecName "kube-api-access-bvhmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:45.411284 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.411154 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvhmt\" (UniqueName: \"kubernetes.io/projected/d9f38975-1b9b-47d0-93c1-0ab25c13d127-kube-api-access-bvhmt\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:45.411284 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.411209 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f38975-1b9b-47d0-93c1-0ab25c13d127-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:45.797781 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.797744 4531 generic.go:356] "Generic (PLEG): container finished" podID="d9f38975-1b9b-47d0-93c1-0ab25c13d127" containerID="4db96304fc3ebc271636b13adcdcd46301d9642c20755393c8c89098c14a4f97" exitCode=0 Oct 30 19:37:45.797975 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.797802 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs" Oct 30 19:37:45.797975 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.797826 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs" event={"ID":"d9f38975-1b9b-47d0-93c1-0ab25c13d127","Type":"ContainerDied","Data":"4db96304fc3ebc271636b13adcdcd46301d9642c20755393c8c89098c14a4f97"} Oct 30 19:37:45.797975 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.797928 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs" event={"ID":"d9f38975-1b9b-47d0-93c1-0ab25c13d127","Type":"ContainerDied","Data":"0cdf548ab478fc43fe9f2e67ec6b922fe2ff7300f3af690f39edadf90bdd3f5b"} Oct 30 19:37:45.797975 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.797964 4531 scope.go:122] "RemoveContainer" containerID="4db96304fc3ebc271636b13adcdcd46301d9642c20755393c8c89098c14a4f97" Oct 30 19:37:45.799394 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.799373 4531 generic.go:356] "Generic (PLEG): container finished" podID="387d077d-6792-45c0-ae7c-2891c00f47c2" containerID="9186cfce414a0a851a046cf9da2260bfa8a94b50215af55b550cf564ad729866" exitCode=0 Oct 30 19:37:45.799512 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.799419 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m" Oct 30 19:37:45.799512 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.799449 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m" event={"ID":"387d077d-6792-45c0-ae7c-2891c00f47c2","Type":"ContainerDied","Data":"9186cfce414a0a851a046cf9da2260bfa8a94b50215af55b550cf564ad729866"} Oct 30 19:37:45.799512 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.799491 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m" event={"ID":"387d077d-6792-45c0-ae7c-2891c00f47c2","Type":"ContainerDied","Data":"79898ad9ce8a5f6e11340684b37c506ba28534ccb6b589929c9b2ca3039d3cd2"} Oct 30 19:37:45.809791 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.809767 4531 scope.go:122] "RemoveContainer" containerID="4db96304fc3ebc271636b13adcdcd46301d9642c20755393c8c89098c14a4f97" Oct 30 19:37:45.810171 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:37:45.810117 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4db96304fc3ebc271636b13adcdcd46301d9642c20755393c8c89098c14a4f97\": container with ID starting with 4db96304fc3ebc271636b13adcdcd46301d9642c20755393c8c89098c14a4f97 not found: ID does not exist" containerID="4db96304fc3ebc271636b13adcdcd46301d9642c20755393c8c89098c14a4f97" Oct 30 19:37:45.810254 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.810183 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4db96304fc3ebc271636b13adcdcd46301d9642c20755393c8c89098c14a4f97"} err="failed to get container status \"4db96304fc3ebc271636b13adcdcd46301d9642c20755393c8c89098c14a4f97\": rpc error: code = NotFound desc = could not find container \"4db96304fc3ebc271636b13adcdcd46301d9642c20755393c8c89098c14a4f97\": container with ID starting with 4db96304fc3ebc271636b13adcdcd46301d9642c20755393c8c89098c14a4f97 not found: ID does not exist" Oct 30 19:37:45.810254 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.810202 4531 scope.go:122] "RemoveContainer" containerID="9186cfce414a0a851a046cf9da2260bfa8a94b50215af55b550cf564ad729866" Oct 30 19:37:45.819706 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.819679 4531 scope.go:122] "RemoveContainer" containerID="9186cfce414a0a851a046cf9da2260bfa8a94b50215af55b550cf564ad729866" Oct 30 19:37:45.820044 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:37:45.820026 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9186cfce414a0a851a046cf9da2260bfa8a94b50215af55b550cf564ad729866\": container with ID starting with 9186cfce414a0a851a046cf9da2260bfa8a94b50215af55b550cf564ad729866 not found: ID does not exist" containerID="9186cfce414a0a851a046cf9da2260bfa8a94b50215af55b550cf564ad729866" Oct 30 19:37:45.820097 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.820053 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9186cfce414a0a851a046cf9da2260bfa8a94b50215af55b550cf564ad729866"} err="failed to get container status \"9186cfce414a0a851a046cf9da2260bfa8a94b50215af55b550cf564ad729866\": rpc error: code = NotFound desc = could not find container \"9186cfce414a0a851a046cf9da2260bfa8a94b50215af55b550cf564ad729866\": container with ID starting with 9186cfce414a0a851a046cf9da2260bfa8a94b50215af55b550cf564ad729866 not found: ID does not exist" Oct 30 19:37:45.967318 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:45.967274 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m"] Oct 30 19:37:46.016543 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:46.016507 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-76a1ef-6746f945bd-s9k6m"] Oct 30 19:37:46.107302 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:46.107176 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="387d077d-6792-45c0-ae7c-2891c00f47c2" path="/var/lib/kubelet/pods/387d077d-6792-45c0-ae7c-2891c00f47c2/volumes" Oct 30 19:37:46.107622 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:46.107601 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs"] Oct 30 19:37:46.107971 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:46.107954 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-76a1ef-6fd6756575-shjbs"] Oct 30 19:37:48.098843 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:48.098809 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9f38975-1b9b-47d0-93c1-0ab25c13d127" path="/var/lib/kubelet/pods/d9f38975-1b9b-47d0-93c1-0ab25c13d127/volumes" Oct 30 19:37:49.350756 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.350694 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7" podUID="1929b710-1a08-4c73-b6fc-ecceac5089f5" containerName="payload-processing" containerID="cri-o://c28bacaf4def14b9fce10f834ab29ab08f7fcbcc32336583559ef4112b37690a" gracePeriod=25 Oct 30 19:37:49.463198 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.463110 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj" podUID="0348ded6-6eb6-4d7b-80b1-aa551aae81eb" containerName="payload-pre-processing" containerID="cri-o://a95d4422bb10ed45d9b450f5aaffdb70a128e78cd8ac0996ce437c3cbc862bba" gracePeriod=25 Oct 30 19:37:49.479815 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:37:49.479792 4531 logging.go:55] [core] [Channel #5111 SubChannel #5112]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.255:50052", ServerName: "10.133.0.255:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.255:50052: connect: connection refused" Oct 30 19:37:49.633403 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.632254 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7" Oct 30 19:37:49.659229 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.659185 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1929b710-1a08-4c73-b6fc-ecceac5089f5-kube-api-access-rwmht\" (UniqueName: \"kubernetes.io/projected/1929b710-1a08-4c73-b6fc-ecceac5089f5-kube-api-access-rwmht\") pod \"1929b710-1a08-4c73-b6fc-ecceac5089f5\" (UID: \"1929b710-1a08-4c73-b6fc-ecceac5089f5\") " Oct 30 19:37:49.659439 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.659386 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1929b710-1a08-4c73-b6fc-ecceac5089f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1929b710-1a08-4c73-b6fc-ecceac5089f5-plugins-config-volume\") pod \"1929b710-1a08-4c73-b6fc-ecceac5089f5\" (UID: \"1929b710-1a08-4c73-b6fc-ecceac5089f5\") " Oct 30 19:37:49.660094 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.660060 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1929b710-1a08-4c73-b6fc-ecceac5089f5-plugins-config-volume" pod "1929b710-1a08-4c73-b6fc-ecceac5089f5" (UID: "1929b710-1a08-4c73-b6fc-ecceac5089f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:49.662927 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.662893 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1929b710-1a08-4c73-b6fc-ecceac5089f5-kube-api-access-rwmht" pod "1929b710-1a08-4c73-b6fc-ecceac5089f5" (UID: "1929b710-1a08-4c73-b6fc-ecceac5089f5"). InnerVolumeSpecName "kube-api-access-rwmht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:49.758367 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.758334 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj" Oct 30 19:37:49.760683 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.760650 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1929b710-1a08-4c73-b6fc-ecceac5089f5-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:49.760683 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.760684 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwmht\" (UniqueName: \"kubernetes.io/projected/1929b710-1a08-4c73-b6fc-ecceac5089f5-kube-api-access-rwmht\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:49.802572 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.802534 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h"] Oct 30 19:37:49.804951 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.804925 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a1ef-rnk6h"] Oct 30 19:37:49.828773 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.828734 4531 generic.go:356] "Generic (PLEG): container finished" podID="0348ded6-6eb6-4d7b-80b1-aa551aae81eb" containerID="a95d4422bb10ed45d9b450f5aaffdb70a128e78cd8ac0996ce437c3cbc862bba" exitCode=0 Oct 30 19:37:49.828920 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.828790 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj" Oct 30 19:37:49.828920 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.828827 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj" event={"ID":"0348ded6-6eb6-4d7b-80b1-aa551aae81eb","Type":"ContainerDied","Data":"a95d4422bb10ed45d9b450f5aaffdb70a128e78cd8ac0996ce437c3cbc862bba"} Oct 30 19:37:49.828920 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.828888 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj" event={"ID":"0348ded6-6eb6-4d7b-80b1-aa551aae81eb","Type":"ContainerDied","Data":"88946e2fcaa93b6c6c8361724b3e78dea4daf2e8628098a34a486d409ed0728a"} Oct 30 19:37:49.828920 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.828915 4531 scope.go:122] "RemoveContainer" containerID="a95d4422bb10ed45d9b450f5aaffdb70a128e78cd8ac0996ce437c3cbc862bba" Oct 30 19:37:49.830514 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.830488 4531 generic.go:356] "Generic (PLEG): container finished" podID="1929b710-1a08-4c73-b6fc-ecceac5089f5" containerID="c28bacaf4def14b9fce10f834ab29ab08f7fcbcc32336583559ef4112b37690a" exitCode=0 Oct 30 19:37:49.830632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.830536 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7" Oct 30 19:37:49.830679 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.830580 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7" event={"ID":"1929b710-1a08-4c73-b6fc-ecceac5089f5","Type":"ContainerDied","Data":"c28bacaf4def14b9fce10f834ab29ab08f7fcbcc32336583559ef4112b37690a"} Oct 30 19:37:49.830750 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.830735 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7" event={"ID":"1929b710-1a08-4c73-b6fc-ecceac5089f5","Type":"ContainerDied","Data":"537b1035892ffddc863fe5bd6398ff7a6ff6f1e26cfa041a273d1aaf7d0b9010"} Oct 30 19:37:49.839647 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.839629 4531 scope.go:122] "RemoveContainer" containerID="a95d4422bb10ed45d9b450f5aaffdb70a128e78cd8ac0996ce437c3cbc862bba" Oct 30 19:37:49.839955 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:37:49.839927 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a95d4422bb10ed45d9b450f5aaffdb70a128e78cd8ac0996ce437c3cbc862bba\": container with ID starting with a95d4422bb10ed45d9b450f5aaffdb70a128e78cd8ac0996ce437c3cbc862bba not found: ID does not exist" containerID="a95d4422bb10ed45d9b450f5aaffdb70a128e78cd8ac0996ce437c3cbc862bba" Oct 30 19:37:49.840050 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.839961 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a95d4422bb10ed45d9b450f5aaffdb70a128e78cd8ac0996ce437c3cbc862bba"} err="failed to get container status \"a95d4422bb10ed45d9b450f5aaffdb70a128e78cd8ac0996ce437c3cbc862bba\": rpc error: code = NotFound desc = could not find container \"a95d4422bb10ed45d9b450f5aaffdb70a128e78cd8ac0996ce437c3cbc862bba\": container with ID starting with a95d4422bb10ed45d9b450f5aaffdb70a128e78cd8ac0996ce437c3cbc862bba not found: ID does not exist" Oct 30 19:37:49.840050 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.839980 4531 scope.go:122] "RemoveContainer" containerID="c28bacaf4def14b9fce10f834ab29ab08f7fcbcc32336583559ef4112b37690a" Oct 30 19:37:49.849734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.849713 4531 scope.go:122] "RemoveContainer" containerID="c28bacaf4def14b9fce10f834ab29ab08f7fcbcc32336583559ef4112b37690a" Oct 30 19:37:49.850037 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:37:49.850017 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c28bacaf4def14b9fce10f834ab29ab08f7fcbcc32336583559ef4112b37690a\": container with ID starting with c28bacaf4def14b9fce10f834ab29ab08f7fcbcc32336583559ef4112b37690a not found: ID does not exist" containerID="c28bacaf4def14b9fce10f834ab29ab08f7fcbcc32336583559ef4112b37690a" Oct 30 19:37:49.850105 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.850047 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c28bacaf4def14b9fce10f834ab29ab08f7fcbcc32336583559ef4112b37690a"} err="failed to get container status \"c28bacaf4def14b9fce10f834ab29ab08f7fcbcc32336583559ef4112b37690a\": rpc error: code = NotFound desc = could not find container \"c28bacaf4def14b9fce10f834ab29ab08f7fcbcc32336583559ef4112b37690a\": container with ID starting with c28bacaf4def14b9fce10f834ab29ab08f7fcbcc32336583559ef4112b37690a not found: ID does not exist" Oct 30 19:37:49.861695 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.861631 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0348ded6-6eb6-4d7b-80b1-aa551aae81eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0348ded6-6eb6-4d7b-80b1-aa551aae81eb-plugins-config-volume\") pod \"0348ded6-6eb6-4d7b-80b1-aa551aae81eb\" (UID: \"0348ded6-6eb6-4d7b-80b1-aa551aae81eb\") " Oct 30 19:37:49.861811 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.861730 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0348ded6-6eb6-4d7b-80b1-aa551aae81eb-kube-api-access-98bhn\" (UniqueName: \"kubernetes.io/projected/0348ded6-6eb6-4d7b-80b1-aa551aae81eb-kube-api-access-98bhn\") pod \"0348ded6-6eb6-4d7b-80b1-aa551aae81eb\" (UID: \"0348ded6-6eb6-4d7b-80b1-aa551aae81eb\") " Oct 30 19:37:49.862014 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.861991 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0348ded6-6eb6-4d7b-80b1-aa551aae81eb-plugins-config-volume" pod "0348ded6-6eb6-4d7b-80b1-aa551aae81eb" (UID: "0348ded6-6eb6-4d7b-80b1-aa551aae81eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:49.864039 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.864015 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0348ded6-6eb6-4d7b-80b1-aa551aae81eb-kube-api-access-98bhn" pod "0348ded6-6eb6-4d7b-80b1-aa551aae81eb" (UID: "0348ded6-6eb6-4d7b-80b1-aa551aae81eb"). InnerVolumeSpecName "kube-api-access-98bhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:49.963481 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.963442 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98bhn\" (UniqueName: \"kubernetes.io/projected/0348ded6-6eb6-4d7b-80b1-aa551aae81eb-kube-api-access-98bhn\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:49.963481 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:49.963472 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0348ded6-6eb6-4d7b-80b1-aa551aae81eb-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:50.049213 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:50.049169 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7"] Oct 30 19:37:50.052298 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:50.052272 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cdd844-84698f5fc8-vqpg7"] Oct 30 19:37:50.107351 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:50.107315 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1929b710-1a08-4c73-b6fc-ecceac5089f5" path="/var/lib/kubelet/pods/1929b710-1a08-4c73-b6fc-ecceac5089f5/volumes" Oct 30 19:37:50.107938 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:50.107916 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a9ce5a7-266b-4e63-8fd5-1630a0bd0931" path="/var/lib/kubelet/pods/5a9ce5a7-266b-4e63-8fd5-1630a0bd0931/volumes" Oct 30 19:37:50.246561 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:50.246521 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj"] Oct 30 19:37:50.250696 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:50.250652 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj"] Oct 30 19:37:50.479491 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:50.479440 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cdd844-7cdd9f88c9-p5skj" podUID="0348ded6-6eb6-4d7b-80b1-aa551aae81eb" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.255:50052\" within 1s: context deadline exceeded" Oct 30 19:37:52.098881 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:52.098840 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0348ded6-6eb6-4d7b-80b1-aa551aae81eb" path="/var/lib/kubelet/pods/0348ded6-6eb6-4d7b-80b1-aa551aae81eb/volumes" Oct 30 19:37:53.541769 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:53.541730 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd"] Oct 30 19:37:53.592240 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:53.592202 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7fdf6b-nh2nd"] Oct 30 19:37:53.795501 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:53.795415 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j"] Oct 30 19:37:54.101791 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:54.101658 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32ae1753-20e4-4947-9bc5-5b52d6a2fbba" path="/var/lib/kubelet/pods/32ae1753-20e4-4947-9bc5-5b52d6a2fbba/volumes" Oct 30 19:37:56.400556 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:37:56.400520 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:37:58.796236 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:58.796196 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j" podUID="c982a4ff-8174-4543-930b-20290e9dcb48" containerName="payload-processing" containerID="cri-o://74cd303636f1cafe17d04bb0b9923357d5bd4237f5cec3a442bf772737ef1cdc" gracePeriod=25 Oct 30 19:37:59.070321 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.070291 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j" Oct 30 19:37:59.168152 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.168094 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c982a4ff-8174-4543-930b-20290e9dcb48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c982a4ff-8174-4543-930b-20290e9dcb48-plugins-config-volume\") pod \"c982a4ff-8174-4543-930b-20290e9dcb48\" (UID: \"c982a4ff-8174-4543-930b-20290e9dcb48\") " Oct 30 19:37:59.168366 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.168207 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c982a4ff-8174-4543-930b-20290e9dcb48-kube-api-access-n5gqn\" (UniqueName: \"kubernetes.io/projected/c982a4ff-8174-4543-930b-20290e9dcb48-kube-api-access-n5gqn\") pod \"c982a4ff-8174-4543-930b-20290e9dcb48\" (UID: \"c982a4ff-8174-4543-930b-20290e9dcb48\") " Oct 30 19:37:59.168502 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.168474 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c982a4ff-8174-4543-930b-20290e9dcb48-plugins-config-volume" pod "c982a4ff-8174-4543-930b-20290e9dcb48" (UID: "c982a4ff-8174-4543-930b-20290e9dcb48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:59.168573 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.168562 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c982a4ff-8174-4543-930b-20290e9dcb48-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:59.170675 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.170647 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c982a4ff-8174-4543-930b-20290e9dcb48-kube-api-access-n5gqn" pod "c982a4ff-8174-4543-930b-20290e9dcb48" (UID: "c982a4ff-8174-4543-930b-20290e9dcb48"). InnerVolumeSpecName "kube-api-access-n5gqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:59.269829 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.269788 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5gqn\" (UniqueName: \"kubernetes.io/projected/c982a4ff-8174-4543-930b-20290e9dcb48-kube-api-access-n5gqn\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:37:59.894861 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.894828 4531 generic.go:356] "Generic (PLEG): container finished" podID="c982a4ff-8174-4543-930b-20290e9dcb48" containerID="74cd303636f1cafe17d04bb0b9923357d5bd4237f5cec3a442bf772737ef1cdc" exitCode=0 Oct 30 19:37:59.895372 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.894885 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j" Oct 30 19:37:59.895372 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.894911 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j" event={"ID":"c982a4ff-8174-4543-930b-20290e9dcb48","Type":"ContainerDied","Data":"74cd303636f1cafe17d04bb0b9923357d5bd4237f5cec3a442bf772737ef1cdc"} Oct 30 19:37:59.895372 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.894975 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j" event={"ID":"c982a4ff-8174-4543-930b-20290e9dcb48","Type":"ContainerDied","Data":"390277e1b237e7af32275c2c9dee71f69d7ac846d194ac80f79292126c6d84bc"} Oct 30 19:37:59.895372 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.895005 4531 scope.go:122] "RemoveContainer" containerID="74cd303636f1cafe17d04bb0b9923357d5bd4237f5cec3a442bf772737ef1cdc" Oct 30 19:37:59.905734 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.905711 4531 scope.go:122] "RemoveContainer" containerID="74cd303636f1cafe17d04bb0b9923357d5bd4237f5cec3a442bf772737ef1cdc" Oct 30 19:37:59.906066 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:37:59.906043 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74cd303636f1cafe17d04bb0b9923357d5bd4237f5cec3a442bf772737ef1cdc\": container with ID starting with 74cd303636f1cafe17d04bb0b9923357d5bd4237f5cec3a442bf772737ef1cdc not found: ID does not exist" containerID="74cd303636f1cafe17d04bb0b9923357d5bd4237f5cec3a442bf772737ef1cdc" Oct 30 19:37:59.906183 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.906076 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74cd303636f1cafe17d04bb0b9923357d5bd4237f5cec3a442bf772737ef1cdc"} err="failed to get container status \"74cd303636f1cafe17d04bb0b9923357d5bd4237f5cec3a442bf772737ef1cdc\": rpc error: code = NotFound desc = could not find container \"74cd303636f1cafe17d04bb0b9923357d5bd4237f5cec3a442bf772737ef1cdc\": container with ID starting with 74cd303636f1cafe17d04bb0b9923357d5bd4237f5cec3a442bf772737ef1cdc not found: ID does not exist" Oct 30 19:37:59.941557 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.941523 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4"] Oct 30 19:37:59.942030 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.942014 4531 cpu_manager.go:414] "containerMap: removing container" podUID="0348ded6-6eb6-4d7b-80b1-aa551aae81eb" containerName="payload-pre-processing" Oct 30 19:37:59.942083 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.942033 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0348ded6-6eb6-4d7b-80b1-aa551aae81eb" containerName="payload-pre-processing" Oct 30 19:37:59.942083 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.942046 4531 cpu_manager.go:414] "containerMap: removing container" podUID="d9f38975-1b9b-47d0-93c1-0ab25c13d127" containerName="payload-pre-processing" Oct 30 19:37:59.942083 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.942054 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9f38975-1b9b-47d0-93c1-0ab25c13d127" containerName="payload-pre-processing" Oct 30 19:37:59.942083 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.942074 4531 cpu_manager.go:414] "containerMap: removing container" podUID="c982a4ff-8174-4543-930b-20290e9dcb48" containerName="payload-processing" Oct 30 19:37:59.942083 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.942079 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c982a4ff-8174-4543-930b-20290e9dcb48" containerName="payload-processing" Oct 30 19:37:59.942274 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.942089 4531 cpu_manager.go:414] "containerMap: removing container" podUID="387d077d-6792-45c0-ae7c-2891c00f47c2" containerName="payload-processing" Oct 30 19:37:59.942274 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.942094 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="387d077d-6792-45c0-ae7c-2891c00f47c2" containerName="payload-processing" Oct 30 19:37:59.942274 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.942117 4531 cpu_manager.go:414] "containerMap: removing container" podUID="1929b710-1a08-4c73-b6fc-ecceac5089f5" containerName="payload-processing" Oct 30 19:37:59.942274 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.942141 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1929b710-1a08-4c73-b6fc-ecceac5089f5" containerName="payload-processing" Oct 30 19:37:59.942274 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.942154 4531 cpu_manager.go:414] "containerMap: removing container" podUID="c428fcd4-f859-416e-913c-4f719d7ded51" containerName="revoke-keys" Oct 30 19:37:59.942274 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.942161 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c428fcd4-f859-416e-913c-4f719d7ded51" containerName="revoke-keys" Oct 30 19:37:59.942274 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.942275 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1929b710-1a08-4c73-b6fc-ecceac5089f5" containerName="payload-processing" Oct 30 19:37:59.942274 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.942284 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c428fcd4-f859-416e-913c-4f719d7ded51" containerName="revoke-keys" Oct 30 19:37:59.942537 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.942293 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0348ded6-6eb6-4d7b-80b1-aa551aae81eb" containerName="payload-pre-processing" Oct 30 19:37:59.942537 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.942299 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9f38975-1b9b-47d0-93c1-0ab25c13d127" containerName="payload-pre-processing" Oct 30 19:37:59.942537 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.942307 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c982a4ff-8174-4543-930b-20290e9dcb48" containerName="payload-processing" Oct 30 19:37:59.942537 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.942318 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="387d077d-6792-45c0-ae7c-2891c00f47c2" containerName="payload-processing" Oct 30 19:37:59.946967 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.946937 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4" Oct 30 19:37:59.987281 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.979439 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1317539-30ac-4327-bf57-2164195f6a44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4\" (UID: \"c1317539-30ac-4327-bf57-2164195f6a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4" Oct 30 19:37:59.987281 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.979577 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1317539-30ac-4327-bf57-2164195f6a44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4\" (UID: \"c1317539-30ac-4327-bf57-2164195f6a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4" Oct 30 19:37:59.987281 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:37:59.979740 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd8dt\" (UniqueName: \"kubernetes.io/projected/c1317539-30ac-4327-bf57-2164195f6a44-kube-api-access-zd8dt\") pod \"maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4\" (UID: \"c1317539-30ac-4327-bf57-2164195f6a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4" Oct 30 19:38:00.001686 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:00.001645 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:38:00.001960 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:00.001939 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:38:00.002756 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:00.002718 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:38:00.002893 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:00.002769 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:38:00.005274 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:00.005243 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4"] Oct 30 19:38:00.081579 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:00.081501 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1317539-30ac-4327-bf57-2164195f6a44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4\" (UID: \"c1317539-30ac-4327-bf57-2164195f6a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4" Oct 30 19:38:00.081579 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:00.081588 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1317539-30ac-4327-bf57-2164195f6a44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4\" (UID: \"c1317539-30ac-4327-bf57-2164195f6a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4" Oct 30 19:38:00.081797 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:00.081625 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd8dt\" (UniqueName: \"kubernetes.io/projected/c1317539-30ac-4327-bf57-2164195f6a44-kube-api-access-zd8dt\") pod \"maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4\" (UID: \"c1317539-30ac-4327-bf57-2164195f6a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4" Oct 30 19:38:00.082249 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:00.082219 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1317539-30ac-4327-bf57-2164195f6a44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4\" (UID: \"c1317539-30ac-4327-bf57-2164195f6a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4" Oct 30 19:38:00.084440 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:00.084411 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1317539-30ac-4327-bf57-2164195f6a44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4\" (UID: \"c1317539-30ac-4327-bf57-2164195f6a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4" Oct 30 19:38:00.198362 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:00.198271 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:38:00.210628 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:00.210596 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd8dt\" (UniqueName: \"kubernetes.io/projected/c1317539-30ac-4327-bf57-2164195f6a44-kube-api-access-zd8dt\") pod \"maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4\" (UID: \"c1317539-30ac-4327-bf57-2164195f6a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4" Oct 30 19:38:00.258353 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:00.258309 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j"] Oct 30 19:38:00.268525 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:00.268478 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-0f83f5-649f9f6cb6-wqr6j"] Oct 30 19:38:00.317509 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:00.317479 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:38:00.321566 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:00.321543 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4" Oct 30 19:38:00.711551 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:00.711368 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4"] Oct 30 19:38:00.714325 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:38:00.714288 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1317539_30ac_4327_bf57_2164195f6a44.slice/crio-db974940cc44807ca7a3b605a3aa0bb407ecffdfec2940d5067afccb2ee69f9b WatchSource:0}: Error finding container db974940cc44807ca7a3b605a3aa0bb407ecffdfec2940d5067afccb2ee69f9b: Status 404 returned error can't find the container with id db974940cc44807ca7a3b605a3aa0bb407ecffdfec2940d5067afccb2ee69f9b Oct 30 19:38:00.903703 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:00.903660 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4" event={"ID":"c1317539-30ac-4327-bf57-2164195f6a44","Type":"ContainerStarted","Data":"db974940cc44807ca7a3b605a3aa0bb407ecffdfec2940d5067afccb2ee69f9b"} Oct 30 19:38:01.913923 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:01.913820 4531 generic.go:356] "Generic (PLEG): container finished" podID="c1317539-30ac-4327-bf57-2164195f6a44" containerID="3918fafdbe1d350982e05c5c76d40aee15c5d626766f79ccd1423f410506e250" exitCode=0 Oct 30 19:38:01.913923 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:01.913905 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4" event={"ID":"c1317539-30ac-4327-bf57-2164195f6a44","Type":"ContainerDied","Data":"3918fafdbe1d350982e05c5c76d40aee15c5d626766f79ccd1423f410506e250"} Oct 30 19:38:02.099499 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:02.099465 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c982a4ff-8174-4543-930b-20290e9dcb48" path="/var/lib/kubelet/pods/c982a4ff-8174-4543-930b-20290e9dcb48/volumes" Oct 30 19:38:03.069030 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.068992 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4" Oct 30 19:38:03.118462 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.118421 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1317539-30ac-4327-bf57-2164195f6a44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1317539-30ac-4327-bf57-2164195f6a44-maas-api-openshift-ca\") pod \"c1317539-30ac-4327-bf57-2164195f6a44\" (UID: \"c1317539-30ac-4327-bf57-2164195f6a44\") " Oct 30 19:38:03.118651 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.118526 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1317539-30ac-4327-bf57-2164195f6a44-kube-api-access-zd8dt\" (UniqueName: \"kubernetes.io/projected/c1317539-30ac-4327-bf57-2164195f6a44-kube-api-access-zd8dt\") pod \"c1317539-30ac-4327-bf57-2164195f6a44\" (UID: \"c1317539-30ac-4327-bf57-2164195f6a44\") " Oct 30 19:38:03.118651 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.118570 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1317539-30ac-4327-bf57-2164195f6a44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1317539-30ac-4327-bf57-2164195f6a44-maas-api-xks-ca\") pod \"c1317539-30ac-4327-bf57-2164195f6a44\" (UID: \"c1317539-30ac-4327-bf57-2164195f6a44\") " Oct 30 19:38:03.118930 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.118902 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1317539-30ac-4327-bf57-2164195f6a44-maas-api-openshift-ca" pod "c1317539-30ac-4327-bf57-2164195f6a44" (UID: "c1317539-30ac-4327-bf57-2164195f6a44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:03.121115 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.121075 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1317539-30ac-4327-bf57-2164195f6a44-maas-api-xks-ca" pod "c1317539-30ac-4327-bf57-2164195f6a44" (UID: "c1317539-30ac-4327-bf57-2164195f6a44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:38:03.121330 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.121214 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1317539-30ac-4327-bf57-2164195f6a44-kube-api-access-zd8dt" pod "c1317539-30ac-4327-bf57-2164195f6a44" (UID: "c1317539-30ac-4327-bf57-2164195f6a44"). InnerVolumeSpecName "kube-api-access-zd8dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:03.121699 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.121678 4531 scope.go:122] "RemoveContainer" containerID="3bb2c31d8bd9882153dd5af9400797f5a6f5922f81390ae66019073efd935974" Oct 30 19:38:03.146184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.146073 4531 scope.go:122] "RemoveContainer" containerID="9acb82373335e605c8b2c1c63ffd179d692737e24b28aff5ad77ffa0a85bb634" Oct 30 19:38:03.156952 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.156921 4531 scope.go:122] "RemoveContainer" containerID="98e0883896194694e2a4b264169f304a417378afeea8b0d10dc1d2e6dd61efa9" Oct 30 19:38:03.173211 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.173111 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc"] Oct 30 19:38:03.173727 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.173710 4531 cpu_manager.go:414] "containerMap: removing container" podUID="c1317539-30ac-4327-bf57-2164195f6a44" containerName="revoke-keys" Oct 30 19:38:03.173770 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.173730 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1317539-30ac-4327-bf57-2164195f6a44" containerName="revoke-keys" Oct 30 19:38:03.173849 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.173840 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1317539-30ac-4327-bf57-2164195f6a44" containerName="revoke-keys" Oct 30 19:38:03.178494 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.178474 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc" Oct 30 19:38:03.219581 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.219552 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1317539-30ac-4327-bf57-2164195f6a44-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:38:03.219581 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.219582 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd8dt\" (UniqueName: \"kubernetes.io/projected/c1317539-30ac-4327-bf57-2164195f6a44-kube-api-access-zd8dt\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:38:03.219581 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.219592 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1317539-30ac-4327-bf57-2164195f6a44-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:38:03.228831 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.228787 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc"] Oct 30 19:38:03.320241 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.320197 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2c2bff8-584a-4947-8795-be0542370b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc\" (UID: \"e2c2bff8-584a-4947-8795-be0542370b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc" Oct 30 19:38:03.320479 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.320249 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trh8c\" (UniqueName: \"kubernetes.io/projected/e2c2bff8-584a-4947-8795-be0542370b6a-kube-api-access-trh8c\") pod \"maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc\" (UID: \"e2c2bff8-584a-4947-8795-be0542370b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc" Oct 30 19:38:03.320553 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.320495 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2c2bff8-584a-4947-8795-be0542370b6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc\" (UID: \"e2c2bff8-584a-4947-8795-be0542370b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc" Oct 30 19:38:03.422077 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.422031 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2c2bff8-584a-4947-8795-be0542370b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc\" (UID: \"e2c2bff8-584a-4947-8795-be0542370b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc" Oct 30 19:38:03.422077 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.422082 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trh8c\" (UniqueName: \"kubernetes.io/projected/e2c2bff8-584a-4947-8795-be0542370b6a-kube-api-access-trh8c\") pod \"maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc\" (UID: \"e2c2bff8-584a-4947-8795-be0542370b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc" Oct 30 19:38:03.422363 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.422215 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2c2bff8-584a-4947-8795-be0542370b6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc\" (UID: \"e2c2bff8-584a-4947-8795-be0542370b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc" Oct 30 19:38:03.422732 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.422711 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2c2bff8-584a-4947-8795-be0542370b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc\" (UID: \"e2c2bff8-584a-4947-8795-be0542370b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc" Oct 30 19:38:03.424884 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.424814 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2c2bff8-584a-4947-8795-be0542370b6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc\" (UID: \"e2c2bff8-584a-4947-8795-be0542370b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc" Oct 30 19:38:03.483526 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.483483 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trh8c\" (UniqueName: \"kubernetes.io/projected/e2c2bff8-584a-4947-8795-be0542370b6a-kube-api-access-trh8c\") pod \"maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc\" (UID: \"e2c2bff8-584a-4947-8795-be0542370b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc" Oct 30 19:38:03.497335 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.497300 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc" Oct 30 19:38:03.880216 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.880051 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc"] Oct 30 19:38:03.883170 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:38:03.883110 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c2bff8_584a_4947_8795_be0542370b6a.slice/crio-4c6e23aa1bf531f3b46399449e31b6c1a1a2e9508d0e667f4a54228b6876b04a WatchSource:0}: Error finding container 4c6e23aa1bf531f3b46399449e31b6c1a1a2e9508d0e667f4a54228b6876b04a: Status 404 returned error can't find the container with id 4c6e23aa1bf531f3b46399449e31b6c1a1a2e9508d0e667f4a54228b6876b04a Oct 30 19:38:03.929103 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.929071 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4" Oct 30 19:38:03.929103 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.929083 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4" event={"ID":"c1317539-30ac-4327-bf57-2164195f6a44","Type":"ContainerDied","Data":"db974940cc44807ca7a3b605a3aa0bb407ecffdfec2940d5067afccb2ee69f9b"} Oct 30 19:38:03.929411 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.929156 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db974940cc44807ca7a3b605a3aa0bb407ecffdfec2940d5067afccb2ee69f9b" Oct 30 19:38:03.930483 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:03.930457 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc" event={"ID":"e2c2bff8-584a-4947-8795-be0542370b6a","Type":"ContainerStarted","Data":"4c6e23aa1bf531f3b46399449e31b6c1a1a2e9508d0e667f4a54228b6876b04a"} Oct 30 19:38:04.382320 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:04.382278 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4"] Oct 30 19:38:04.938536 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:04.938500 4531 generic.go:356] "Generic (PLEG): container finished" podID="e2c2bff8-584a-4947-8795-be0542370b6a" containerID="f166c9d34832d373f1591b5651bfa413a9c0f7447394ace9a6eafafcd4e8bbaf" exitCode=0 Oct 30 19:38:04.938733 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:04.938591 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc" event={"ID":"e2c2bff8-584a-4947-8795-be0542370b6a","Type":"ContainerDied","Data":"f166c9d34832d373f1591b5651bfa413a9c0f7447394ace9a6eafafcd4e8bbaf"} Oct 30 19:38:06.101139 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:06.101105 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc" Oct 30 19:38:06.256050 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:06.256010 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2c2bff8-584a-4947-8795-be0542370b6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2c2bff8-584a-4947-8795-be0542370b6a-maas-api-xks-ca\") pod \"e2c2bff8-584a-4947-8795-be0542370b6a\" (UID: \"e2c2bff8-584a-4947-8795-be0542370b6a\") " Oct 30 19:38:06.256303 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:06.256199 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2c2bff8-584a-4947-8795-be0542370b6a-kube-api-access-trh8c\" (UniqueName: \"kubernetes.io/projected/e2c2bff8-584a-4947-8795-be0542370b6a-kube-api-access-trh8c\") pod \"e2c2bff8-584a-4947-8795-be0542370b6a\" (UID: \"e2c2bff8-584a-4947-8795-be0542370b6a\") " Oct 30 19:38:06.256303 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:06.256268 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2c2bff8-584a-4947-8795-be0542370b6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2c2bff8-584a-4947-8795-be0542370b6a-maas-api-openshift-ca\") pod \"e2c2bff8-584a-4947-8795-be0542370b6a\" (UID: \"e2c2bff8-584a-4947-8795-be0542370b6a\") " Oct 30 19:38:06.256710 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:06.256678 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2c2bff8-584a-4947-8795-be0542370b6a-maas-api-openshift-ca" pod "e2c2bff8-584a-4947-8795-be0542370b6a" (UID: "e2c2bff8-584a-4947-8795-be0542370b6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:06.259017 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:06.258987 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2c2bff8-584a-4947-8795-be0542370b6a-kube-api-access-trh8c" pod "e2c2bff8-584a-4947-8795-be0542370b6a" (UID: "e2c2bff8-584a-4947-8795-be0542370b6a"). InnerVolumeSpecName "kube-api-access-trh8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:06.259233 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:06.259100 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2c2bff8-584a-4947-8795-be0542370b6a-maas-api-xks-ca" pod "e2c2bff8-584a-4947-8795-be0542370b6a" (UID: "e2c2bff8-584a-4947-8795-be0542370b6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:38:06.358298 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:06.358253 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trh8c\" (UniqueName: \"kubernetes.io/projected/e2c2bff8-584a-4947-8795-be0542370b6a-kube-api-access-trh8c\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:38:06.358298 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:06.358296 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2c2bff8-584a-4947-8795-be0542370b6a-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:38:06.358533 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:06.358310 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2c2bff8-584a-4947-8795-be0542370b6a-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:38:06.952618 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:06.952584 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc" Oct 30 19:38:06.952618 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:06.952605 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc" event={"ID":"e2c2bff8-584a-4947-8795-be0542370b6a","Type":"ContainerDied","Data":"4c6e23aa1bf531f3b46399449e31b6c1a1a2e9508d0e667f4a54228b6876b04a"} Oct 30 19:38:06.952857 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:06.952654 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c6e23aa1bf531f3b46399449e31b6c1a1a2e9508d0e667f4a54228b6876b04a" Oct 30 19:38:14.212812 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:14.212779 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k"] Oct 30 19:38:19.213956 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.213912 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k" podUID="99261072-41ec-47ab-976f-969826789970" containerName="payload-processing" containerID="cri-o://8532d3300dd437d76cbf20f3eca2ba814f49dcb640f1f548281ae4839312744e" gracePeriod=25 Oct 30 19:38:19.403637 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.403582 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" podUID="1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630" containerName="main" containerID="cri-o://ade2531a8928518ceaf2b4926c679e0606e48fa045224293bbe51f10610c682c" gracePeriod=45 Oct 30 19:38:19.504317 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.504272 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" podUID="1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": dial tcp 10.133.0.37:8000: connect: connection refused" Oct 30 19:38:19.534402 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.534364 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k" Oct 30 19:38:19.608909 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.608880 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99261072-41ec-47ab-976f-969826789970-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99261072-41ec-47ab-976f-969826789970-plugins-config-volume\") pod \"99261072-41ec-47ab-976f-969826789970\" (UID: \"99261072-41ec-47ab-976f-969826789970\") " Oct 30 19:38:19.609060 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.609047 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99261072-41ec-47ab-976f-969826789970-kube-api-access-2lsbs\" (UniqueName: \"kubernetes.io/projected/99261072-41ec-47ab-976f-969826789970-kube-api-access-2lsbs\") pod \"99261072-41ec-47ab-976f-969826789970\" (UID: \"99261072-41ec-47ab-976f-969826789970\") " Oct 30 19:38:19.609302 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.609279 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99261072-41ec-47ab-976f-969826789970-plugins-config-volume" pod "99261072-41ec-47ab-976f-969826789970" (UID: "99261072-41ec-47ab-976f-969826789970"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:19.609412 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.609398 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99261072-41ec-47ab-976f-969826789970-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:38:19.611589 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.611546 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99261072-41ec-47ab-976f-969826789970-kube-api-access-2lsbs" pod "99261072-41ec-47ab-976f-969826789970" (UID: "99261072-41ec-47ab-976f-969826789970"). InnerVolumeSpecName "kube-api-access-2lsbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:19.678883 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.678852 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 19:38:19.710626 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.710572 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-home\" (UniqueName: \"kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-home\") pod \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\" (UID: \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\") " Oct 30 19:38:19.710834 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.710648 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-kube-api-access-l4r5d\" (UniqueName: \"kubernetes.io/projected/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-kube-api-access-l4r5d\") pod \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\" (UID: \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\") " Oct 30 19:38:19.710834 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.710708 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-model-cache\") pod \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\" (UID: \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\") " Oct 30 19:38:19.710834 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.710757 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-tls-certs\" (UniqueName: \"kubernetes.io/secret/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-tls-certs\") pod \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\" (UID: \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\") " Oct 30 19:38:19.711023 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.710836 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-tmp-dir\") pod \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\" (UID: \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\") " Oct 30 19:38:19.711023 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.710881 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-dshm\") pod \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\" (UID: \"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630\") " Oct 30 19:38:19.711023 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.710907 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-home" pod "1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630" (UID: "1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:19.711255 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.711006 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-model-cache" pod "1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630" (UID: "1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:19.711255 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.711071 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-tmp-dir" pod "1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630" (UID: "1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:19.711440 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.711411 4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-home\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:38:19.711440 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.711439 4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-model-cache\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:38:19.711805 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.711451 4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-tmp-dir\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:38:19.711805 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.711465 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lsbs\" (UniqueName: \"kubernetes.io/projected/99261072-41ec-47ab-976f-969826789970-kube-api-access-2lsbs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:38:19.713427 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.713394 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-tls-certs" pod "1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630" (UID: "1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:38:19.713569 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.713426 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-kube-api-access-l4r5d" pod "1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630" (UID: "1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630"). InnerVolumeSpecName "kube-api-access-l4r5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:19.713569 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.713501 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-dshm" pod "1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630" (UID: "1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:19.813293 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.813245 4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-tls-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:38:19.813293 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.813289 4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-dshm\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:38:19.813613 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:19.813304 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4r5d\" (UniqueName: \"kubernetes.io/projected/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630-kube-api-access-l4r5d\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:38:20.037290 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:20.037251 4531 generic.go:356] "Generic (PLEG): container finished" podID="99261072-41ec-47ab-976f-969826789970" containerID="8532d3300dd437d76cbf20f3eca2ba814f49dcb640f1f548281ae4839312744e" exitCode=0 Oct 30 19:38:20.037873 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:20.037301 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k" Oct 30 19:38:20.037873 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:20.037360 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k" event={"ID":"99261072-41ec-47ab-976f-969826789970","Type":"ContainerDied","Data":"8532d3300dd437d76cbf20f3eca2ba814f49dcb640f1f548281ae4839312744e"} Oct 30 19:38:20.037873 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:20.037414 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k" event={"ID":"99261072-41ec-47ab-976f-969826789970","Type":"ContainerDied","Data":"3e780aff814466982f7e10a3770b756574835464135bb0e248b138c61223a74e"} Oct 30 19:38:20.037873 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:20.037445 4531 scope.go:122] "RemoveContainer" containerID="8532d3300dd437d76cbf20f3eca2ba814f49dcb640f1f548281ae4839312744e" Oct 30 19:38:20.041099 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:20.041069 4531 generic.go:356] "Generic (PLEG): container finished" podID="1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630" containerID="ade2531a8928518ceaf2b4926c679e0606e48fa045224293bbe51f10610c682c" exitCode=0 Oct 30 19:38:20.041272 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:20.041142 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" Oct 30 19:38:20.041272 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:20.041185 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" event={"ID":"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630","Type":"ContainerDied","Data":"ade2531a8928518ceaf2b4926c679e0606e48fa045224293bbe51f10610c682c"} Oct 30 19:38:20.041374 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:20.041271 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4" event={"ID":"1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630","Type":"ContainerDied","Data":"3d26c25d0015a77c69d07f92e2e855ab481a3aa6659f85ed9d867c2748022bff"} Oct 30 19:38:20.054379 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:20.054250 4531 scope.go:122] "RemoveContainer" containerID="8532d3300dd437d76cbf20f3eca2ba814f49dcb640f1f548281ae4839312744e" Oct 30 19:38:20.054895 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:38:20.054860 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8532d3300dd437d76cbf20f3eca2ba814f49dcb640f1f548281ae4839312744e\": container with ID starting with 8532d3300dd437d76cbf20f3eca2ba814f49dcb640f1f548281ae4839312744e not found: ID does not exist" containerID="8532d3300dd437d76cbf20f3eca2ba814f49dcb640f1f548281ae4839312744e" Oct 30 19:38:20.055067 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:20.054910 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8532d3300dd437d76cbf20f3eca2ba814f49dcb640f1f548281ae4839312744e"} err="failed to get container status \"8532d3300dd437d76cbf20f3eca2ba814f49dcb640f1f548281ae4839312744e\": rpc error: code = NotFound desc = could not find container \"8532d3300dd437d76cbf20f3eca2ba814f49dcb640f1f548281ae4839312744e\": container with ID starting with 8532d3300dd437d76cbf20f3eca2ba814f49dcb640f1f548281ae4839312744e not found: ID does not exist" Oct 30 19:38:20.055067 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:20.054936 4531 scope.go:122] "RemoveContainer" containerID="ade2531a8928518ceaf2b4926c679e0606e48fa045224293bbe51f10610c682c" Oct 30 19:38:20.067653 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:20.067622 4531 scope.go:122] "RemoveContainer" containerID="ade2531a8928518ceaf2b4926c679e0606e48fa045224293bbe51f10610c682c" Oct 30 19:38:20.068101 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:38:20.068072 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ade2531a8928518ceaf2b4926c679e0606e48fa045224293bbe51f10610c682c\": container with ID starting with ade2531a8928518ceaf2b4926c679e0606e48fa045224293bbe51f10610c682c not found: ID does not exist" containerID="ade2531a8928518ceaf2b4926c679e0606e48fa045224293bbe51f10610c682c" Oct 30 19:38:20.068239 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:20.068117 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ade2531a8928518ceaf2b4926c679e0606e48fa045224293bbe51f10610c682c"} err="failed to get container status \"ade2531a8928518ceaf2b4926c679e0606e48fa045224293bbe51f10610c682c\": rpc error: code = NotFound desc = could not find container \"ade2531a8928518ceaf2b4926c679e0606e48fa045224293bbe51f10610c682c\": container with ID starting with ade2531a8928518ceaf2b4926c679e0606e48fa045224293bbe51f10610c682c not found: ID does not exist" Oct 30 19:38:20.206485 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:20.206454 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4"] Oct 30 19:38:20.210320 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:20.210288 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3b6013/simulator-w3-3b6013-kserve-74476cc78c-kvzz4"] Oct 30 19:38:20.319463 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:20.319374 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k"] Oct 30 19:38:20.323946 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:20.323905 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-185e8b-67789b7ffb-hkv9k"] Oct 30 19:38:22.098389 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:22.098353 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630" path="/var/lib/kubelet/pods/1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630/volumes" Oct 30 19:38:22.098854 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:22.098762 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99261072-41ec-47ab-976f-969826789970" path="/var/lib/kubelet/pods/99261072-41ec-47ab-976f-969826789970/volumes" Oct 30 19:38:26.399645 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:38:26.399606 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:38:41.149577 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.149542 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv"] Oct 30 19:38:41.150074 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.150034 4531 cpu_manager.go:414] "containerMap: removing container" podUID="e2c2bff8-584a-4947-8795-be0542370b6a" containerName="revoke-keys" Oct 30 19:38:41.150074 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.150049 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2c2bff8-584a-4947-8795-be0542370b6a" containerName="revoke-keys" Oct 30 19:38:41.150074 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.150074 4531 cpu_manager.go:414] "containerMap: removing container" podUID="1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630" containerName="main" Oct 30 19:38:41.150210 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.150080 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630" containerName="main" Oct 30 19:38:41.150210 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.150094 4531 cpu_manager.go:414] "containerMap: removing container" podUID="99261072-41ec-47ab-976f-969826789970" containerName="payload-processing" Oct 30 19:38:41.150210 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.150099 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99261072-41ec-47ab-976f-969826789970" containerName="payload-processing" Oct 30 19:38:41.150310 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.150220 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2c2bff8-584a-4947-8795-be0542370b6a" containerName="revoke-keys" Oct 30 19:38:41.150310 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.150230 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b32fdf8-e824-4cc6-b25b-ef5f1ed2d630" containerName="main" Oct 30 19:38:41.150310 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.150238 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="99261072-41ec-47ab-976f-969826789970" containerName="payload-processing" Oct 30 19:38:41.153602 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.153581 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv" Oct 30 19:38:41.211519 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.211480 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:38:41.211519 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.211479 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:38:41.215164 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.215101 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv"] Oct 30 19:38:41.244730 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.244690 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ebe169-0a70-4fc5-8bed-7b7570987600-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv\" (UID: \"43ebe169-0a70-4fc5-8bed-7b7570987600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv" Oct 30 19:38:41.244967 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.244752 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwsdv\" (UniqueName: \"kubernetes.io/projected/43ebe169-0a70-4fc5-8bed-7b7570987600-kube-api-access-dwsdv\") pod \"maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv\" (UID: \"43ebe169-0a70-4fc5-8bed-7b7570987600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv" Oct 30 19:38:41.244967 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.244854 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ebe169-0a70-4fc5-8bed-7b7570987600-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv\" (UID: \"43ebe169-0a70-4fc5-8bed-7b7570987600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv" Oct 30 19:38:41.260973 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.260927 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:38:41.261207 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.261003 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:38:41.345656 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.345618 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ebe169-0a70-4fc5-8bed-7b7570987600-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv\" (UID: \"43ebe169-0a70-4fc5-8bed-7b7570987600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv" Oct 30 19:38:41.345877 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.345668 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwsdv\" (UniqueName: \"kubernetes.io/projected/43ebe169-0a70-4fc5-8bed-7b7570987600-kube-api-access-dwsdv\") pod \"maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv\" (UID: \"43ebe169-0a70-4fc5-8bed-7b7570987600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv" Oct 30 19:38:41.345877 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.345804 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ebe169-0a70-4fc5-8bed-7b7570987600-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv\" (UID: \"43ebe169-0a70-4fc5-8bed-7b7570987600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv" Oct 30 19:38:41.346514 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.346487 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ebe169-0a70-4fc5-8bed-7b7570987600-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv\" (UID: \"43ebe169-0a70-4fc5-8bed-7b7570987600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv" Oct 30 19:38:41.348884 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.348861 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ebe169-0a70-4fc5-8bed-7b7570987600-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv\" (UID: \"43ebe169-0a70-4fc5-8bed-7b7570987600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv" Oct 30 19:38:41.408565 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.408490 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwsdv\" (UniqueName: \"kubernetes.io/projected/43ebe169-0a70-4fc5-8bed-7b7570987600-kube-api-access-dwsdv\") pod \"maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv\" (UID: \"43ebe169-0a70-4fc5-8bed-7b7570987600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv" Oct 30 19:38:41.469005 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.468957 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv" Oct 30 19:38:41.862567 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:41.862537 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv"] Oct 30 19:38:41.864785 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:38:41.864746 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ebe169_0a70_4fc5_8bed_7b7570987600.slice/crio-7c2b6b9a34e818e3c708ddd23e32f216355fe1c7204164c1e14bca238abb8a22 WatchSource:0}: Error finding container 7c2b6b9a34e818e3c708ddd23e32f216355fe1c7204164c1e14bca238abb8a22: Status 404 returned error can't find the container with id 7c2b6b9a34e818e3c708ddd23e32f216355fe1c7204164c1e14bca238abb8a22 Oct 30 19:38:42.171943 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:42.171900 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv" event={"ID":"43ebe169-0a70-4fc5-8bed-7b7570987600","Type":"ContainerStarted","Data":"9f2b6cf475d3d2bc24a59726b19dc21a3ee83c68425a01632a736e9ccb3fa290"} Oct 30 19:38:42.172378 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:42.171973 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv" event={"ID":"43ebe169-0a70-4fc5-8bed-7b7570987600","Type":"ContainerStarted","Data":"7c2b6b9a34e818e3c708ddd23e32f216355fe1c7204164c1e14bca238abb8a22"} Oct 30 19:38:42.300708 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:42.300647 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv" podStartSLOduration=1.300630846 podStartE2EDuration="1.300630846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:42.297721944 +0000 UTC m=+3882.324826903" watchObservedRunningTime="2026-10-30 19:38:42.300630846 +0000 UTC m=+3882.327735804" Oct 30 19:38:43.196537 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:43.196493 4531 generic.go:356] "Generic (PLEG): container finished" podID="43ebe169-0a70-4fc5-8bed-7b7570987600" containerID="9f2b6cf475d3d2bc24a59726b19dc21a3ee83c68425a01632a736e9ccb3fa290" exitCode=0 Oct 30 19:38:43.196955 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:43.196549 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv" event={"ID":"43ebe169-0a70-4fc5-8bed-7b7570987600","Type":"ContainerDied","Data":"9f2b6cf475d3d2bc24a59726b19dc21a3ee83c68425a01632a736e9ccb3fa290"} Oct 30 19:38:44.358812 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:44.358778 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv" Oct 30 19:38:44.482312 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:44.482212 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ebe169-0a70-4fc5-8bed-7b7570987600-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ebe169-0a70-4fc5-8bed-7b7570987600-maas-api-xks-ca\") pod \"43ebe169-0a70-4fc5-8bed-7b7570987600\" (UID: \"43ebe169-0a70-4fc5-8bed-7b7570987600\") " Oct 30 19:38:44.482498 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:44.482356 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ebe169-0a70-4fc5-8bed-7b7570987600-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ebe169-0a70-4fc5-8bed-7b7570987600-maas-api-openshift-ca\") pod \"43ebe169-0a70-4fc5-8bed-7b7570987600\" (UID: \"43ebe169-0a70-4fc5-8bed-7b7570987600\") " Oct 30 19:38:44.482498 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:44.482472 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ebe169-0a70-4fc5-8bed-7b7570987600-kube-api-access-dwsdv\" (UniqueName: \"kubernetes.io/projected/43ebe169-0a70-4fc5-8bed-7b7570987600-kube-api-access-dwsdv\") pod \"43ebe169-0a70-4fc5-8bed-7b7570987600\" (UID: \"43ebe169-0a70-4fc5-8bed-7b7570987600\") " Oct 30 19:38:44.482759 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:44.482727 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ebe169-0a70-4fc5-8bed-7b7570987600-maas-api-openshift-ca" pod "43ebe169-0a70-4fc5-8bed-7b7570987600" (UID: "43ebe169-0a70-4fc5-8bed-7b7570987600"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:44.482916 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:44.482895 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ebe169-0a70-4fc5-8bed-7b7570987600-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:38:44.485322 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:44.485292 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ebe169-0a70-4fc5-8bed-7b7570987600-maas-api-xks-ca" pod "43ebe169-0a70-4fc5-8bed-7b7570987600" (UID: "43ebe169-0a70-4fc5-8bed-7b7570987600"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:38:44.485441 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:44.485321 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ebe169-0a70-4fc5-8bed-7b7570987600-kube-api-access-dwsdv" pod "43ebe169-0a70-4fc5-8bed-7b7570987600" (UID: "43ebe169-0a70-4fc5-8bed-7b7570987600"). InnerVolumeSpecName "kube-api-access-dwsdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:44.584628 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:44.584584 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ebe169-0a70-4fc5-8bed-7b7570987600-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:38:44.584628 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:44.584622 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwsdv\" (UniqueName: \"kubernetes.io/projected/43ebe169-0a70-4fc5-8bed-7b7570987600-kube-api-access-dwsdv\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:38:45.209729 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:45.209698 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv" Oct 30 19:38:45.209921 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:45.209728 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv" event={"ID":"43ebe169-0a70-4fc5-8bed-7b7570987600","Type":"ContainerDied","Data":"7c2b6b9a34e818e3c708ddd23e32f216355fe1c7204164c1e14bca238abb8a22"} Oct 30 19:38:45.209921 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:45.209779 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c2b6b9a34e818e3c708ddd23e32f216355fe1c7204164c1e14bca238abb8a22" Oct 30 19:38:56.398113 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:38:56.398070 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:38:59.141866 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.141825 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afd486d1-j4s24"] Oct 30 19:38:59.142537 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.142516 4531 cpu_manager.go:414] "containerMap: removing container" podUID="43ebe169-0a70-4fc5-8bed-7b7570987600" containerName="revoke-keys" Oct 30 19:38:59.142584 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.142539 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ebe169-0a70-4fc5-8bed-7b7570987600" containerName="revoke-keys" Oct 30 19:38:59.142663 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.142653 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ebe169-0a70-4fc5-8bed-7b7570987600" containerName="revoke-keys" Oct 30 19:38:59.145979 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.145958 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd486d1-j4s24" Oct 30 19:38:59.218002 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.217955 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:38:59.218002 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.217987 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:38:59.219960 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.219932 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afd486d1-j4s24"] Oct 30 19:38:59.235411 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.234893 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:38:59.236188 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.236166 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:38:59.243620 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.243596 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znpwm\" (UniqueName: \"kubernetes.io/projected/00856537-9db1-4d00-867d-c409322f77d4-kube-api-access-znpwm\") pod \"maas-api-revoke-sub-afd486d1-j4s24\" (UID: \"00856537-9db1-4d00-867d-c409322f77d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd486d1-j4s24" Oct 30 19:38:59.243748 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.243730 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00856537-9db1-4d00-867d-c409322f77d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afd486d1-j4s24\" (UID: \"00856537-9db1-4d00-867d-c409322f77d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd486d1-j4s24" Oct 30 19:38:59.243876 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.243860 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00856537-9db1-4d00-867d-c409322f77d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afd486d1-j4s24\" (UID: \"00856537-9db1-4d00-867d-c409322f77d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd486d1-j4s24" Oct 30 19:38:59.248423 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.248400 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21979b84-4sckl"] Oct 30 19:38:59.252190 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.252172 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21979b84-4sckl" Oct 30 19:38:59.320698 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.320498 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21979b84-4sckl"] Oct 30 19:38:59.344755 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.344717 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00856537-9db1-4d00-867d-c409322f77d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afd486d1-j4s24\" (UID: \"00856537-9db1-4d00-867d-c409322f77d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd486d1-j4s24" Oct 30 19:38:59.344967 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.344777 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7pzh\" (UniqueName: \"kubernetes.io/projected/06da2630-8109-40ed-b33d-9051092f06c7-kube-api-access-d7pzh\") pod \"maas-api-revoke-sub-21979b84-4sckl\" (UID: \"06da2630-8109-40ed-b33d-9051092f06c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21979b84-4sckl" Oct 30 19:38:59.344967 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.344816 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znpwm\" (UniqueName: \"kubernetes.io/projected/00856537-9db1-4d00-867d-c409322f77d4-kube-api-access-znpwm\") pod \"maas-api-revoke-sub-afd486d1-j4s24\" (UID: \"00856537-9db1-4d00-867d-c409322f77d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd486d1-j4s24" Oct 30 19:38:59.344967 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.344933 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00856537-9db1-4d00-867d-c409322f77d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afd486d1-j4s24\" (UID: \"00856537-9db1-4d00-867d-c409322f77d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd486d1-j4s24" Oct 30 19:38:59.345182 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.344976 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06da2630-8109-40ed-b33d-9051092f06c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21979b84-4sckl\" (UID: \"06da2630-8109-40ed-b33d-9051092f06c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21979b84-4sckl" Oct 30 19:38:59.345182 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.345006 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06da2630-8109-40ed-b33d-9051092f06c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21979b84-4sckl\" (UID: \"06da2630-8109-40ed-b33d-9051092f06c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21979b84-4sckl" Oct 30 19:38:59.345737 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.345705 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00856537-9db1-4d00-867d-c409322f77d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afd486d1-j4s24\" (UID: \"00856537-9db1-4d00-867d-c409322f77d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd486d1-j4s24" Oct 30 19:38:59.348318 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.348288 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00856537-9db1-4d00-867d-c409322f77d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afd486d1-j4s24\" (UID: \"00856537-9db1-4d00-867d-c409322f77d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd486d1-j4s24" Oct 30 19:38:59.408064 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.407983 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znpwm\" (UniqueName: \"kubernetes.io/projected/00856537-9db1-4d00-867d-c409322f77d4-kube-api-access-znpwm\") pod \"maas-api-revoke-sub-afd486d1-j4s24\" (UID: \"00856537-9db1-4d00-867d-c409322f77d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd486d1-j4s24" Oct 30 19:38:59.446460 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.446425 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06da2630-8109-40ed-b33d-9051092f06c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21979b84-4sckl\" (UID: \"06da2630-8109-40ed-b33d-9051092f06c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21979b84-4sckl" Oct 30 19:38:59.446460 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.446464 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06da2630-8109-40ed-b33d-9051092f06c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21979b84-4sckl\" (UID: \"06da2630-8109-40ed-b33d-9051092f06c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21979b84-4sckl" Oct 30 19:38:59.446718 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.446537 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7pzh\" (UniqueName: \"kubernetes.io/projected/06da2630-8109-40ed-b33d-9051092f06c7-kube-api-access-d7pzh\") pod \"maas-api-revoke-sub-21979b84-4sckl\" (UID: \"06da2630-8109-40ed-b33d-9051092f06c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21979b84-4sckl" Oct 30 19:38:59.447305 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.447278 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06da2630-8109-40ed-b33d-9051092f06c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21979b84-4sckl\" (UID: \"06da2630-8109-40ed-b33d-9051092f06c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21979b84-4sckl" Oct 30 19:38:59.449263 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.449233 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06da2630-8109-40ed-b33d-9051092f06c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21979b84-4sckl\" (UID: \"06da2630-8109-40ed-b33d-9051092f06c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21979b84-4sckl" Oct 30 19:38:59.459446 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.459424 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd486d1-j4s24" Oct 30 19:38:59.509692 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.509658 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7pzh\" (UniqueName: \"kubernetes.io/projected/06da2630-8109-40ed-b33d-9051092f06c7-kube-api-access-d7pzh\") pod \"maas-api-revoke-sub-21979b84-4sckl\" (UID: \"06da2630-8109-40ed-b33d-9051092f06c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21979b84-4sckl" Oct 30 19:38:59.566648 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.566602 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21979b84-4sckl" Oct 30 19:38:59.852708 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.852668 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afd486d1-j4s24"] Oct 30 19:38:59.853422 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:38:59.853388 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00856537_9db1_4d00_867d_c409322f77d4.slice/crio-7f7ba7d04bd2091238de1334241cf362805cfff852675a38ea0cef8e9baed070 WatchSource:0}: Error finding container 7f7ba7d04bd2091238de1334241cf362805cfff852675a38ea0cef8e9baed070: Status 404 returned error can't find the container with id 7f7ba7d04bd2091238de1334241cf362805cfff852675a38ea0cef8e9baed070 Oct 30 19:38:59.951816 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:38:59.951791 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21979b84-4sckl"] Oct 30 19:38:59.953845 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:38:59.953819 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06da2630_8109_40ed_b33d_9051092f06c7.slice/crio-3ad2360f66396d1d5c9a9d65ad7d43f0b33eea7cfadb67d04b7cb17ef3836a67 WatchSource:0}: Error finding container 3ad2360f66396d1d5c9a9d65ad7d43f0b33eea7cfadb67d04b7cb17ef3836a67: Status 404 returned error can't find the container with id 3ad2360f66396d1d5c9a9d65ad7d43f0b33eea7cfadb67d04b7cb17ef3836a67 Oct 30 19:39:00.298462 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:00.298424 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21979b84-4sckl" event={"ID":"06da2630-8109-40ed-b33d-9051092f06c7","Type":"ContainerStarted","Data":"a12bf608280cacb5171a4c1a24939098f248ff0c5ac9bb46e7a98a930d4182fb"} Oct 30 19:39:00.298933 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:00.298490 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21979b84-4sckl" event={"ID":"06da2630-8109-40ed-b33d-9051092f06c7","Type":"ContainerStarted","Data":"3ad2360f66396d1d5c9a9d65ad7d43f0b33eea7cfadb67d04b7cb17ef3836a67"} Oct 30 19:39:00.300167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:00.300120 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd486d1-j4s24" event={"ID":"00856537-9db1-4d00-867d-c409322f77d4","Type":"ContainerStarted","Data":"99b9fba606b1e3a8a2ee795d58dd725920986c688c400be19b98740f7b2af7dc"} Oct 30 19:39:00.300297 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:00.300183 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd486d1-j4s24" event={"ID":"00856537-9db1-4d00-867d-c409322f77d4","Type":"ContainerStarted","Data":"7f7ba7d04bd2091238de1334241cf362805cfff852675a38ea0cef8e9baed070"} Oct 30 19:39:00.419728 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:00.419626 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21979b84-4sckl" podStartSLOduration=1.419609914 podStartE2EDuration="1.419609914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:00.416413017 +0000 UTC m=+3900.443518166" watchObservedRunningTime="2026-10-30 19:39:00.419609914 +0000 UTC m=+3900.446714918" Oct 30 19:39:00.537485 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:00.537415 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd486d1-j4s24" podStartSLOduration=1.537398179 podStartE2EDuration="1.537398179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:00.535996115 +0000 UTC m=+3900.563101073" watchObservedRunningTime="2026-10-30 19:39:00.537398179 +0000 UTC m=+3900.564503139" Oct 30 19:39:01.307237 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:01.307202 4531 generic.go:356] "Generic (PLEG): container finished" podID="06da2630-8109-40ed-b33d-9051092f06c7" containerID="a12bf608280cacb5171a4c1a24939098f248ff0c5ac9bb46e7a98a930d4182fb" exitCode=0 Oct 30 19:39:01.307715 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:01.307290 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21979b84-4sckl" event={"ID":"06da2630-8109-40ed-b33d-9051092f06c7","Type":"ContainerDied","Data":"a12bf608280cacb5171a4c1a24939098f248ff0c5ac9bb46e7a98a930d4182fb"} Oct 30 19:39:01.309052 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:01.309027 4531 generic.go:356] "Generic (PLEG): container finished" podID="00856537-9db1-4d00-867d-c409322f77d4" containerID="99b9fba606b1e3a8a2ee795d58dd725920986c688c400be19b98740f7b2af7dc" exitCode=0 Oct 30 19:39:01.309206 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:01.309092 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd486d1-j4s24" event={"ID":"00856537-9db1-4d00-867d-c409322f77d4","Type":"ContainerDied","Data":"99b9fba606b1e3a8a2ee795d58dd725920986c688c400be19b98740f7b2af7dc"} Oct 30 19:39:01.642397 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:01.642297 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm"] Oct 30 19:39:01.648424 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:01.648382 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cdd844-br4qm"] Oct 30 19:39:02.099075 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:02.099035 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5c0dff6-715a-4d21-a4a6-185976b86774" path="/var/lib/kubelet/pods/d5c0dff6-715a-4d21-a4a6-185976b86774/volumes" Oct 30 19:39:02.462852 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:02.462823 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21979b84-4sckl" Oct 30 19:39:02.505488 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:02.505454 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd486d1-j4s24" Oct 30 19:39:02.585653 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:02.585548 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00856537-9db1-4d00-867d-c409322f77d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00856537-9db1-4d00-867d-c409322f77d4-maas-api-xks-ca\") pod \"00856537-9db1-4d00-867d-c409322f77d4\" (UID: \"00856537-9db1-4d00-867d-c409322f77d4\") " Oct 30 19:39:02.585653 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:02.585618 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06da2630-8109-40ed-b33d-9051092f06c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06da2630-8109-40ed-b33d-9051092f06c7-maas-api-openshift-ca\") pod \"06da2630-8109-40ed-b33d-9051092f06c7\" (UID: \"06da2630-8109-40ed-b33d-9051092f06c7\") " Oct 30 19:39:02.585653 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:02.585658 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00856537-9db1-4d00-867d-c409322f77d4-kube-api-access-znpwm\" (UniqueName: \"kubernetes.io/projected/00856537-9db1-4d00-867d-c409322f77d4-kube-api-access-znpwm\") pod \"00856537-9db1-4d00-867d-c409322f77d4\" (UID: \"00856537-9db1-4d00-867d-c409322f77d4\") " Oct 30 19:39:02.586028 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:02.585690 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00856537-9db1-4d00-867d-c409322f77d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00856537-9db1-4d00-867d-c409322f77d4-maas-api-openshift-ca\") pod \"00856537-9db1-4d00-867d-c409322f77d4\" (UID: \"00856537-9db1-4d00-867d-c409322f77d4\") " Oct 30 19:39:02.586028 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:02.585725 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06da2630-8109-40ed-b33d-9051092f06c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06da2630-8109-40ed-b33d-9051092f06c7-maas-api-xks-ca\") pod \"06da2630-8109-40ed-b33d-9051092f06c7\" (UID: \"06da2630-8109-40ed-b33d-9051092f06c7\") " Oct 30 19:39:02.586028 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:02.585761 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06da2630-8109-40ed-b33d-9051092f06c7-kube-api-access-d7pzh\" (UniqueName: \"kubernetes.io/projected/06da2630-8109-40ed-b33d-9051092f06c7-kube-api-access-d7pzh\") pod \"06da2630-8109-40ed-b33d-9051092f06c7\" (UID: \"06da2630-8109-40ed-b33d-9051092f06c7\") " Oct 30 19:39:02.586714 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:02.586676 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00856537-9db1-4d00-867d-c409322f77d4-maas-api-openshift-ca" pod "00856537-9db1-4d00-867d-c409322f77d4" (UID: "00856537-9db1-4d00-867d-c409322f77d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:39:02.586855 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:02.586741 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06da2630-8109-40ed-b33d-9051092f06c7-maas-api-openshift-ca" pod "06da2630-8109-40ed-b33d-9051092f06c7" (UID: "06da2630-8109-40ed-b33d-9051092f06c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:39:02.589215 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:02.589183 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06da2630-8109-40ed-b33d-9051092f06c7-maas-api-xks-ca" pod "06da2630-8109-40ed-b33d-9051092f06c7" (UID: "06da2630-8109-40ed-b33d-9051092f06c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:39:02.589337 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:02.589215 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06da2630-8109-40ed-b33d-9051092f06c7-kube-api-access-d7pzh" pod "06da2630-8109-40ed-b33d-9051092f06c7" (UID: "06da2630-8109-40ed-b33d-9051092f06c7"). InnerVolumeSpecName "kube-api-access-d7pzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:02.589517 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:02.589493 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00856537-9db1-4d00-867d-c409322f77d4-kube-api-access-znpwm" pod "00856537-9db1-4d00-867d-c409322f77d4" (UID: "00856537-9db1-4d00-867d-c409322f77d4"). InnerVolumeSpecName "kube-api-access-znpwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:02.590083 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:02.590055 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00856537-9db1-4d00-867d-c409322f77d4-maas-api-xks-ca" pod "00856537-9db1-4d00-867d-c409322f77d4" (UID: "00856537-9db1-4d00-867d-c409322f77d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:39:02.687355 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:02.687259 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00856537-9db1-4d00-867d-c409322f77d4-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:39:02.687355 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:02.687298 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06da2630-8109-40ed-b33d-9051092f06c7-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:39:02.687355 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:02.687308 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znpwm\" (UniqueName: \"kubernetes.io/projected/00856537-9db1-4d00-867d-c409322f77d4-kube-api-access-znpwm\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:39:02.687355 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:02.687319 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00856537-9db1-4d00-867d-c409322f77d4-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:39:02.687355 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:02.687327 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06da2630-8109-40ed-b33d-9051092f06c7-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:39:02.687355 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:02.687335 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7pzh\" (UniqueName: \"kubernetes.io/projected/06da2630-8109-40ed-b33d-9051092f06c7-kube-api-access-d7pzh\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:39:03.329736 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:03.329708 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd486d1-j4s24" Oct 30 19:39:03.329940 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:03.329742 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd486d1-j4s24" event={"ID":"00856537-9db1-4d00-867d-c409322f77d4","Type":"ContainerDied","Data":"7f7ba7d04bd2091238de1334241cf362805cfff852675a38ea0cef8e9baed070"} Oct 30 19:39:03.329940 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:03.329797 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f7ba7d04bd2091238de1334241cf362805cfff852675a38ea0cef8e9baed070" Oct 30 19:39:03.331349 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:03.331327 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21979b84-4sckl" Oct 30 19:39:03.331349 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:03.331340 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21979b84-4sckl" event={"ID":"06da2630-8109-40ed-b33d-9051092f06c7","Type":"ContainerDied","Data":"3ad2360f66396d1d5c9a9d65ad7d43f0b33eea7cfadb67d04b7cb17ef3836a67"} Oct 30 19:39:03.331541 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:03.331392 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ad2360f66396d1d5c9a9d65ad7d43f0b33eea7cfadb67d04b7cb17ef3836a67" Oct 30 19:39:08.269243 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:08.269195 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b"] Oct 30 19:39:08.318260 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:08.318220 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0f83f5-vfc4b"] Oct 30 19:39:10.098258 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:10.098226 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c428fcd4-f859-416e-913c-4f719d7ded51" path="/var/lib/kubelet/pods/c428fcd4-f859-416e-913c-4f719d7ded51/volumes" Oct 30 19:39:11.342481 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:11.342393 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd"] Oct 30 19:39:11.342933 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:11.342890 4531 cpu_manager.go:414] "containerMap: removing container" podUID="06da2630-8109-40ed-b33d-9051092f06c7" containerName="revoke-keys" Oct 30 19:39:11.342933 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:11.342904 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06da2630-8109-40ed-b33d-9051092f06c7" containerName="revoke-keys" Oct 30 19:39:11.342933 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:11.342928 4531 cpu_manager.go:414] "containerMap: removing container" podUID="00856537-9db1-4d00-867d-c409322f77d4" containerName="revoke-keys" Oct 30 19:39:11.342933 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:11.342934 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00856537-9db1-4d00-867d-c409322f77d4" containerName="revoke-keys" Oct 30 19:39:11.343083 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:11.343039 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="06da2630-8109-40ed-b33d-9051092f06c7" containerName="revoke-keys" Oct 30 19:39:11.343083 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:11.343052 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="00856537-9db1-4d00-867d-c409322f77d4" containerName="revoke-keys" Oct 30 19:39:11.345875 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:11.345854 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd" Oct 30 19:39:11.398459 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:11.398419 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:39:11.398647 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:11.398496 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:39:11.448107 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:11.448078 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:39:11.448349 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:11.448111 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:39:11.451054 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:11.451024 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd"] Oct 30 19:39:11.485123 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:11.485073 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n75n\" (UniqueName: \"kubernetes.io/projected/5c283bdf-4f79-45c2-bef8-b95f716d2aff-kube-api-access-6n75n\") pod \"maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd\" (UID: \"5c283bdf-4f79-45c2-bef8-b95f716d2aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd" Oct 30 19:39:11.485348 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:11.485228 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c283bdf-4f79-45c2-bef8-b95f716d2aff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd\" (UID: \"5c283bdf-4f79-45c2-bef8-b95f716d2aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd" Oct 30 19:39:11.485348 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:11.485260 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c283bdf-4f79-45c2-bef8-b95f716d2aff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd\" (UID: \"5c283bdf-4f79-45c2-bef8-b95f716d2aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd" Oct 30 19:39:11.586784 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:11.586739 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n75n\" (UniqueName: \"kubernetes.io/projected/5c283bdf-4f79-45c2-bef8-b95f716d2aff-kube-api-access-6n75n\") pod \"maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd\" (UID: \"5c283bdf-4f79-45c2-bef8-b95f716d2aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd" Oct 30 19:39:11.586987 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:11.586809 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c283bdf-4f79-45c2-bef8-b95f716d2aff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd\" (UID: \"5c283bdf-4f79-45c2-bef8-b95f716d2aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd" Oct 30 19:39:11.586987 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:11.586838 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c283bdf-4f79-45c2-bef8-b95f716d2aff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd\" (UID: \"5c283bdf-4f79-45c2-bef8-b95f716d2aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd" Oct 30 19:39:11.587548 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:11.587523 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c283bdf-4f79-45c2-bef8-b95f716d2aff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd\" (UID: \"5c283bdf-4f79-45c2-bef8-b95f716d2aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd" Oct 30 19:39:11.589397 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:11.589375 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c283bdf-4f79-45c2-bef8-b95f716d2aff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd\" (UID: \"5c283bdf-4f79-45c2-bef8-b95f716d2aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd" Oct 30 19:39:11.655315 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:11.655233 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n75n\" (UniqueName: \"kubernetes.io/projected/5c283bdf-4f79-45c2-bef8-b95f716d2aff-kube-api-access-6n75n\") pod \"maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd\" (UID: \"5c283bdf-4f79-45c2-bef8-b95f716d2aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd" Oct 30 19:39:11.661940 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:11.661915 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd" Oct 30 19:39:12.048661 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:12.048633 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd"] Oct 30 19:39:12.051589 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:39:12.051558 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c283bdf_4f79_45c2_bef8_b95f716d2aff.slice/crio-d544ab38cedb1ce0bd10b5f7c3e7569dc85228d0a9dc6707be27ddcf4092025e WatchSource:0}: Error finding container d544ab38cedb1ce0bd10b5f7c3e7569dc85228d0a9dc6707be27ddcf4092025e: Status 404 returned error can't find the container with id d544ab38cedb1ce0bd10b5f7c3e7569dc85228d0a9dc6707be27ddcf4092025e Oct 30 19:39:12.384285 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:12.384239 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd" event={"ID":"5c283bdf-4f79-45c2-bef8-b95f716d2aff","Type":"ContainerStarted","Data":"e65f32bb3870462a6658005e75d0d840280d817b01f388ade36e5a070809104c"} Oct 30 19:39:12.384686 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:12.384298 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd" event={"ID":"5c283bdf-4f79-45c2-bef8-b95f716d2aff","Type":"ContainerStarted","Data":"d544ab38cedb1ce0bd10b5f7c3e7569dc85228d0a9dc6707be27ddcf4092025e"} Oct 30 19:39:12.506663 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:12.503151 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd" podStartSLOduration=1.503113528 podStartE2EDuration="1.503113528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:12.501006757 +0000 UTC m=+3912.528111721" watchObservedRunningTime="2026-10-30 19:39:12.503113528 +0000 UTC m=+3912.530218488" Oct 30 19:39:13.391473 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:13.391438 4531 generic.go:356] "Generic (PLEG): container finished" podID="5c283bdf-4f79-45c2-bef8-b95f716d2aff" containerID="e65f32bb3870462a6658005e75d0d840280d817b01f388ade36e5a070809104c" exitCode=0 Oct 30 19:39:13.391950 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:13.391534 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd" event={"ID":"5c283bdf-4f79-45c2-bef8-b95f716d2aff","Type":"ContainerDied","Data":"e65f32bb3870462a6658005e75d0d840280d817b01f388ade36e5a070809104c"} Oct 30 19:39:14.545521 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:14.545496 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd" Oct 30 19:39:14.725003 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:14.724908 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c283bdf-4f79-45c2-bef8-b95f716d2aff-kube-api-access-6n75n\" (UniqueName: \"kubernetes.io/projected/5c283bdf-4f79-45c2-bef8-b95f716d2aff-kube-api-access-6n75n\") pod \"5c283bdf-4f79-45c2-bef8-b95f716d2aff\" (UID: \"5c283bdf-4f79-45c2-bef8-b95f716d2aff\") " Oct 30 19:39:14.725209 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:14.725073 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c283bdf-4f79-45c2-bef8-b95f716d2aff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c283bdf-4f79-45c2-bef8-b95f716d2aff-maas-api-xks-ca\") pod \"5c283bdf-4f79-45c2-bef8-b95f716d2aff\" (UID: \"5c283bdf-4f79-45c2-bef8-b95f716d2aff\") " Oct 30 19:39:14.725278 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:14.725204 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c283bdf-4f79-45c2-bef8-b95f716d2aff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c283bdf-4f79-45c2-bef8-b95f716d2aff-maas-api-openshift-ca\") pod \"5c283bdf-4f79-45c2-bef8-b95f716d2aff\" (UID: \"5c283bdf-4f79-45c2-bef8-b95f716d2aff\") " Oct 30 19:39:14.725574 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:14.725548 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c283bdf-4f79-45c2-bef8-b95f716d2aff-maas-api-openshift-ca" pod "5c283bdf-4f79-45c2-bef8-b95f716d2aff" (UID: "5c283bdf-4f79-45c2-bef8-b95f716d2aff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:39:14.727381 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:14.727346 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c283bdf-4f79-45c2-bef8-b95f716d2aff-maas-api-xks-ca" pod "5c283bdf-4f79-45c2-bef8-b95f716d2aff" (UID: "5c283bdf-4f79-45c2-bef8-b95f716d2aff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:39:14.727496 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:14.727411 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c283bdf-4f79-45c2-bef8-b95f716d2aff-kube-api-access-6n75n" pod "5c283bdf-4f79-45c2-bef8-b95f716d2aff" (UID: "5c283bdf-4f79-45c2-bef8-b95f716d2aff"). InnerVolumeSpecName "kube-api-access-6n75n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:14.826315 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:14.826274 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c283bdf-4f79-45c2-bef8-b95f716d2aff-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:39:14.826315 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:14.826307 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c283bdf-4f79-45c2-bef8-b95f716d2aff-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:39:14.826315 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:14.826321 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n75n\" (UniqueName: \"kubernetes.io/projected/5c283bdf-4f79-45c2-bef8-b95f716d2aff-kube-api-access-6n75n\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:39:15.404546 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:15.404507 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd" Oct 30 19:39:15.404546 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:15.404538 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd" event={"ID":"5c283bdf-4f79-45c2-bef8-b95f716d2aff","Type":"ContainerDied","Data":"d544ab38cedb1ce0bd10b5f7c3e7569dc85228d0a9dc6707be27ddcf4092025e"} Oct 30 19:39:15.404846 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:15.404590 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d544ab38cedb1ce0bd10b5f7c3e7569dc85228d0a9dc6707be27ddcf4092025e" Oct 30 19:39:22.950277 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:22.950243 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv"] Oct 30 19:39:23.059663 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:23.059633 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4"] Oct 30 19:39:25.312950 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:25.312900 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc"] Oct 30 19:39:25.362355 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:25.362311 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-185e8b-4bxzc"] Oct 30 19:39:26.098649 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:26.098614 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2c2bff8-584a-4947-8795-be0542370b6a" path="/var/lib/kubelet/pods/e2c2bff8-584a-4947-8795-be0542370b6a/volumes" Oct 30 19:39:26.398251 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:39:26.398163 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:39:27.950839 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:27.950794 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv" podUID="47e3267f-ce2e-4d73-b3d6-ac88d5316300" containerName="payload-processing" containerID="cri-o://59a54abfea72103028ba6a69d012e9717fd25c092683c5a3ed51922b7ba49298" gracePeriod=25 Oct 30 19:39:28.060332 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.060278 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4" podUID="89fa9ca7-786c-4d84-abce-46b6e81f87bc" containerName="payload-pre-processing" containerID="cri-o://fce2c87a6445ddcebeee09fc921c9efd077f3bd92727e321b07476143194e22d" gracePeriod=25 Oct 30 19:39:28.213572 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.213501 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv" Oct 30 19:39:28.271084 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.271057 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e3267f-ce2e-4d73-b3d6-ac88d5316300-kube-api-access-sd72v\" (UniqueName: \"kubernetes.io/projected/47e3267f-ce2e-4d73-b3d6-ac88d5316300-kube-api-access-sd72v\") pod \"47e3267f-ce2e-4d73-b3d6-ac88d5316300\" (UID: \"47e3267f-ce2e-4d73-b3d6-ac88d5316300\") " Oct 30 19:39:28.271271 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.271105 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47e3267f-ce2e-4d73-b3d6-ac88d5316300-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e3267f-ce2e-4d73-b3d6-ac88d5316300-plugins-config-volume\") pod \"47e3267f-ce2e-4d73-b3d6-ac88d5316300\" (UID: \"47e3267f-ce2e-4d73-b3d6-ac88d5316300\") " Oct 30 19:39:28.271534 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.271514 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e3267f-ce2e-4d73-b3d6-ac88d5316300-plugins-config-volume" pod "47e3267f-ce2e-4d73-b3d6-ac88d5316300" (UID: "47e3267f-ce2e-4d73-b3d6-ac88d5316300"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:39:28.273514 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.273490 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e3267f-ce2e-4d73-b3d6-ac88d5316300-kube-api-access-sd72v" pod "47e3267f-ce2e-4d73-b3d6-ac88d5316300" (UID: "47e3267f-ce2e-4d73-b3d6-ac88d5316300"). InnerVolumeSpecName "kube-api-access-sd72v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:28.308668 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.308645 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4" Oct 30 19:39:28.372444 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.372405 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89fa9ca7-786c-4d84-abce-46b6e81f87bc-kube-api-access-kh4gl\" (UniqueName: \"kubernetes.io/projected/89fa9ca7-786c-4d84-abce-46b6e81f87bc-kube-api-access-kh4gl\") pod \"89fa9ca7-786c-4d84-abce-46b6e81f87bc\" (UID: \"89fa9ca7-786c-4d84-abce-46b6e81f87bc\") " Oct 30 19:39:28.372646 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.372549 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89fa9ca7-786c-4d84-abce-46b6e81f87bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89fa9ca7-786c-4d84-abce-46b6e81f87bc-plugins-config-volume\") pod \"89fa9ca7-786c-4d84-abce-46b6e81f87bc\" (UID: \"89fa9ca7-786c-4d84-abce-46b6e81f87bc\") " Oct 30 19:39:28.373020 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.372983 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89fa9ca7-786c-4d84-abce-46b6e81f87bc-plugins-config-volume" pod "89fa9ca7-786c-4d84-abce-46b6e81f87bc" (UID: "89fa9ca7-786c-4d84-abce-46b6e81f87bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:39:28.373348 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.373022 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd72v\" (UniqueName: \"kubernetes.io/projected/47e3267f-ce2e-4d73-b3d6-ac88d5316300-kube-api-access-sd72v\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:39:28.373348 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.373044 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e3267f-ce2e-4d73-b3d6-ac88d5316300-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:39:28.374842 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.374820 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89fa9ca7-786c-4d84-abce-46b6e81f87bc-kube-api-access-kh4gl" pod "89fa9ca7-786c-4d84-abce-46b6e81f87bc" (UID: "89fa9ca7-786c-4d84-abce-46b6e81f87bc"). InnerVolumeSpecName "kube-api-access-kh4gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:28.473995 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.473918 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89fa9ca7-786c-4d84-abce-46b6e81f87bc-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:39:28.473995 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.473946 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh4gl\" (UniqueName: \"kubernetes.io/projected/89fa9ca7-786c-4d84-abce-46b6e81f87bc-kube-api-access-kh4gl\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:39:28.485932 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.485895 4531 generic.go:356] "Generic (PLEG): container finished" podID="89fa9ca7-786c-4d84-abce-46b6e81f87bc" containerID="fce2c87a6445ddcebeee09fc921c9efd077f3bd92727e321b07476143194e22d" exitCode=0 Oct 30 19:39:28.486166 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.485961 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4" Oct 30 19:39:28.486166 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.485975 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4" event={"ID":"89fa9ca7-786c-4d84-abce-46b6e81f87bc","Type":"ContainerDied","Data":"fce2c87a6445ddcebeee09fc921c9efd077f3bd92727e321b07476143194e22d"} Oct 30 19:39:28.486166 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.486034 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4" event={"ID":"89fa9ca7-786c-4d84-abce-46b6e81f87bc","Type":"ContainerDied","Data":"509418ef264c110d8fe0f189b45bf8009a132a5cebcd4cb3fcec17b830c6fb90"} Oct 30 19:39:28.486166 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.486064 4531 scope.go:122] "RemoveContainer" containerID="fce2c87a6445ddcebeee09fc921c9efd077f3bd92727e321b07476143194e22d" Oct 30 19:39:28.487530 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.487508 4531 generic.go:356] "Generic (PLEG): container finished" podID="47e3267f-ce2e-4d73-b3d6-ac88d5316300" containerID="59a54abfea72103028ba6a69d012e9717fd25c092683c5a3ed51922b7ba49298" exitCode=0 Oct 30 19:39:28.487640 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.487554 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv" Oct 30 19:39:28.487640 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.487580 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv" event={"ID":"47e3267f-ce2e-4d73-b3d6-ac88d5316300","Type":"ContainerDied","Data":"59a54abfea72103028ba6a69d012e9717fd25c092683c5a3ed51922b7ba49298"} Oct 30 19:39:28.487640 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.487624 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv" event={"ID":"47e3267f-ce2e-4d73-b3d6-ac88d5316300","Type":"ContainerDied","Data":"f39092dcccbb9594fb5d1b00c2db9006c564bc09408fd4d313113649674d1e0c"} Oct 30 19:39:28.498556 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.498524 4531 scope.go:122] "RemoveContainer" containerID="fce2c87a6445ddcebeee09fc921c9efd077f3bd92727e321b07476143194e22d" Oct 30 19:39:28.499225 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:39:28.499184 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce2c87a6445ddcebeee09fc921c9efd077f3bd92727e321b07476143194e22d\": container with ID starting with fce2c87a6445ddcebeee09fc921c9efd077f3bd92727e321b07476143194e22d not found: ID does not exist" containerID="fce2c87a6445ddcebeee09fc921c9efd077f3bd92727e321b07476143194e22d" Oct 30 19:39:28.499377 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.499237 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce2c87a6445ddcebeee09fc921c9efd077f3bd92727e321b07476143194e22d"} err="failed to get container status \"fce2c87a6445ddcebeee09fc921c9efd077f3bd92727e321b07476143194e22d\": rpc error: code = NotFound desc = could not find container \"fce2c87a6445ddcebeee09fc921c9efd077f3bd92727e321b07476143194e22d\": container with ID starting with fce2c87a6445ddcebeee09fc921c9efd077f3bd92727e321b07476143194e22d not found: ID does not exist" Oct 30 19:39:28.499377 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.499265 4531 scope.go:122] "RemoveContainer" containerID="59a54abfea72103028ba6a69d012e9717fd25c092683c5a3ed51922b7ba49298" Oct 30 19:39:28.510399 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.510372 4531 scope.go:122] "RemoveContainer" containerID="59a54abfea72103028ba6a69d012e9717fd25c092683c5a3ed51922b7ba49298" Oct 30 19:39:28.510779 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:39:28.510747 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59a54abfea72103028ba6a69d012e9717fd25c092683c5a3ed51922b7ba49298\": container with ID starting with 59a54abfea72103028ba6a69d012e9717fd25c092683c5a3ed51922b7ba49298 not found: ID does not exist" containerID="59a54abfea72103028ba6a69d012e9717fd25c092683c5a3ed51922b7ba49298" Oct 30 19:39:28.510841 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.510779 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59a54abfea72103028ba6a69d012e9717fd25c092683c5a3ed51922b7ba49298"} err="failed to get container status \"59a54abfea72103028ba6a69d012e9717fd25c092683c5a3ed51922b7ba49298\": rpc error: code = NotFound desc = could not find container \"59a54abfea72103028ba6a69d012e9717fd25c092683c5a3ed51922b7ba49298\": container with ID starting with 59a54abfea72103028ba6a69d012e9717fd25c092683c5a3ed51922b7ba49298 not found: ID does not exist" Oct 30 19:39:28.650051 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.650011 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4"] Oct 30 19:39:28.658483 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.658447 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-25e5ef-7b8c77f999-fbpk4"] Oct 30 19:39:28.767780 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.767731 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv"] Oct 30 19:39:28.775947 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:28.775908 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-25e5ef-79ffdd5b8b-j27xv"] Oct 30 19:39:30.106315 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:30.106262 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47e3267f-ce2e-4d73-b3d6-ac88d5316300" path="/var/lib/kubelet/pods/47e3267f-ce2e-4d73-b3d6-ac88d5316300/volumes" Oct 30 19:39:30.106733 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:30.106696 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89fa9ca7-786c-4d84-abce-46b6e81f87bc" path="/var/lib/kubelet/pods/89fa9ca7-786c-4d84-abce-46b6e81f87bc/volumes" Oct 30 19:39:34.059568 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.059518 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc"] Oct 30 19:39:34.060062 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.059991 4531 cpu_manager.go:414] "containerMap: removing container" podUID="5c283bdf-4f79-45c2-bef8-b95f716d2aff" containerName="revoke-keys" Oct 30 19:39:34.060062 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.060004 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c283bdf-4f79-45c2-bef8-b95f716d2aff" containerName="revoke-keys" Oct 30 19:39:34.060062 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.060022 4531 cpu_manager.go:414] "containerMap: removing container" podUID="47e3267f-ce2e-4d73-b3d6-ac88d5316300" containerName="payload-processing" Oct 30 19:39:34.060062 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.060029 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e3267f-ce2e-4d73-b3d6-ac88d5316300" containerName="payload-processing" Oct 30 19:39:34.060062 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.060052 4531 cpu_manager.go:414] "containerMap: removing container" podUID="89fa9ca7-786c-4d84-abce-46b6e81f87bc" containerName="payload-pre-processing" Oct 30 19:39:34.060062 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.060058 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89fa9ca7-786c-4d84-abce-46b6e81f87bc" containerName="payload-pre-processing" Oct 30 19:39:34.060405 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.060138 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c283bdf-4f79-45c2-bef8-b95f716d2aff" containerName="revoke-keys" Oct 30 19:39:34.060405 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.060147 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e3267f-ce2e-4d73-b3d6-ac88d5316300" containerName="payload-processing" Oct 30 19:39:34.060405 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.060158 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="89fa9ca7-786c-4d84-abce-46b6e81f87bc" containerName="payload-pre-processing" Oct 30 19:39:34.064856 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.064836 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc" Oct 30 19:39:34.134431 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.134404 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:39:34.134932 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.134917 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:39:34.145546 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.145509 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb8hp\" (UniqueName: \"kubernetes.io/projected/ba782ca9-28c4-4df0-b467-b727d2b1c70b-kube-api-access-hb8hp\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc\" (UID: \"ba782ca9-28c4-4df0-b467-b727d2b1c70b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc" Oct 30 19:39:34.146018 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.145987 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba782ca9-28c4-4df0-b467-b727d2b1c70b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc\" (UID: \"ba782ca9-28c4-4df0-b467-b727d2b1c70b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc" Oct 30 19:39:34.146171 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.146032 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba782ca9-28c4-4df0-b467-b727d2b1c70b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc\" (UID: \"ba782ca9-28c4-4df0-b467-b727d2b1c70b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc" Oct 30 19:39:34.184658 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.184627 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:39:34.187053 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.187033 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc"] Oct 30 19:39:34.234769 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.234743 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:39:34.247353 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.247329 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba782ca9-28c4-4df0-b467-b727d2b1c70b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc\" (UID: \"ba782ca9-28c4-4df0-b467-b727d2b1c70b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc" Oct 30 19:39:34.247431 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.247360 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba782ca9-28c4-4df0-b467-b727d2b1c70b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc\" (UID: \"ba782ca9-28c4-4df0-b467-b727d2b1c70b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc" Oct 30 19:39:34.247431 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.247404 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb8hp\" (UniqueName: \"kubernetes.io/projected/ba782ca9-28c4-4df0-b467-b727d2b1c70b-kube-api-access-hb8hp\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc\" (UID: \"ba782ca9-28c4-4df0-b467-b727d2b1c70b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc" Oct 30 19:39:34.247919 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.247901 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba782ca9-28c4-4df0-b467-b727d2b1c70b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc\" (UID: \"ba782ca9-28c4-4df0-b467-b727d2b1c70b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc" Oct 30 19:39:34.249899 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.249878 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba782ca9-28c4-4df0-b467-b727d2b1c70b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc\" (UID: \"ba782ca9-28c4-4df0-b467-b727d2b1c70b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc" Oct 30 19:39:34.311300 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.311235 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb8hp\" (UniqueName: \"kubernetes.io/projected/ba782ca9-28c4-4df0-b467-b727d2b1c70b-kube-api-access-hb8hp\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc\" (UID: \"ba782ca9-28c4-4df0-b467-b727d2b1c70b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc" Oct 30 19:39:34.377183 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.377120 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc" Oct 30 19:39:34.757156 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:34.757105 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc"] Oct 30 19:39:34.759482 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:39:34.759452 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba782ca9_28c4_4df0_b467_b727d2b1c70b.slice/crio-94fe7570fa3b93cb77aea04eb7d2588ff10f75e65bd9727969b7b20f49995b46 WatchSource:0}: Error finding container 94fe7570fa3b93cb77aea04eb7d2588ff10f75e65bd9727969b7b20f49995b46: Status 404 returned error can't find the container with id 94fe7570fa3b93cb77aea04eb7d2588ff10f75e65bd9727969b7b20f49995b46 Oct 30 19:39:35.532418 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:35.532388 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc" event={"ID":"ba782ca9-28c4-4df0-b467-b727d2b1c70b","Type":"ContainerStarted","Data":"4e203e12f839eb16b6b255d466a5ecb1bc42dff3f7ae64edf9a991432609daa8"} Oct 30 19:39:35.532817 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:35.532438 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc" event={"ID":"ba782ca9-28c4-4df0-b467-b727d2b1c70b","Type":"ContainerStarted","Data":"94fe7570fa3b93cb77aea04eb7d2588ff10f75e65bd9727969b7b20f49995b46"} Oct 30 19:39:35.649417 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:35.649359 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc" podStartSLOduration=2.649344901 podStartE2EDuration="2.649344901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:35.647421378 +0000 UTC m=+3935.674526350" watchObservedRunningTime="2026-10-30 19:39:35.649344901 +0000 UTC m=+3935.676449860" Oct 30 19:39:36.540753 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:36.540716 4531 generic.go:356] "Generic (PLEG): container finished" podID="ba782ca9-28c4-4df0-b467-b727d2b1c70b" containerID="4e203e12f839eb16b6b255d466a5ecb1bc42dff3f7ae64edf9a991432609daa8" exitCode=0 Oct 30 19:39:36.541230 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:36.540801 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc" event={"ID":"ba782ca9-28c4-4df0-b467-b727d2b1c70b","Type":"ContainerDied","Data":"4e203e12f839eb16b6b255d466a5ecb1bc42dff3f7ae64edf9a991432609daa8"} Oct 30 19:39:37.698527 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:37.698498 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc" Oct 30 19:39:37.788292 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:37.788253 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba782ca9-28c4-4df0-b467-b727d2b1c70b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba782ca9-28c4-4df0-b467-b727d2b1c70b-maas-api-xks-ca\") pod \"ba782ca9-28c4-4df0-b467-b727d2b1c70b\" (UID: \"ba782ca9-28c4-4df0-b467-b727d2b1c70b\") " Oct 30 19:39:37.788481 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:37.788348 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba782ca9-28c4-4df0-b467-b727d2b1c70b-kube-api-access-hb8hp\" (UniqueName: \"kubernetes.io/projected/ba782ca9-28c4-4df0-b467-b727d2b1c70b-kube-api-access-hb8hp\") pod \"ba782ca9-28c4-4df0-b467-b727d2b1c70b\" (UID: \"ba782ca9-28c4-4df0-b467-b727d2b1c70b\") " Oct 30 19:39:37.788481 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:37.788388 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba782ca9-28c4-4df0-b467-b727d2b1c70b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba782ca9-28c4-4df0-b467-b727d2b1c70b-maas-api-openshift-ca\") pod \"ba782ca9-28c4-4df0-b467-b727d2b1c70b\" (UID: \"ba782ca9-28c4-4df0-b467-b727d2b1c70b\") " Oct 30 19:39:37.788816 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:37.788794 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba782ca9-28c4-4df0-b467-b727d2b1c70b-maas-api-openshift-ca" pod "ba782ca9-28c4-4df0-b467-b727d2b1c70b" (UID: "ba782ca9-28c4-4df0-b467-b727d2b1c70b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:39:37.790593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:37.790566 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba782ca9-28c4-4df0-b467-b727d2b1c70b-maas-api-xks-ca" pod "ba782ca9-28c4-4df0-b467-b727d2b1c70b" (UID: "ba782ca9-28c4-4df0-b467-b727d2b1c70b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:39:37.790705 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:37.790662 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba782ca9-28c4-4df0-b467-b727d2b1c70b-kube-api-access-hb8hp" pod "ba782ca9-28c4-4df0-b467-b727d2b1c70b" (UID: "ba782ca9-28c4-4df0-b467-b727d2b1c70b"). InnerVolumeSpecName "kube-api-access-hb8hp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:37.889881 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:37.889842 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb8hp\" (UniqueName: \"kubernetes.io/projected/ba782ca9-28c4-4df0-b467-b727d2b1c70b-kube-api-access-hb8hp\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:39:37.889881 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:37.889879 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba782ca9-28c4-4df0-b467-b727d2b1c70b-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:39:37.889881 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:37.889890 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba782ca9-28c4-4df0-b467-b727d2b1c70b-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:39:38.553167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:38.553121 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc" Oct 30 19:39:38.553386 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:38.553162 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc" event={"ID":"ba782ca9-28c4-4df0-b467-b727d2b1c70b","Type":"ContainerDied","Data":"94fe7570fa3b93cb77aea04eb7d2588ff10f75e65bd9727969b7b20f49995b46"} Oct 30 19:39:38.553386 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:39:38.553216 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94fe7570fa3b93cb77aea04eb7d2588ff10f75e65bd9727969b7b20f49995b46" Oct 30 19:39:56.399994 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:39:56.399951 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:40:01.512557 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:01.512513 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv"] Oct 30 19:40:01.516460 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:01.516427 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9e4e63-hshsv"] Oct 30 19:40:02.098897 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:02.098850 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43ebe169-0a70-4fc5-8bed-7b7570987600" path="/var/lib/kubelet/pods/43ebe169-0a70-4fc5-8bed-7b7570987600/volumes" Oct 30 19:40:09.682113 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:09.682072 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2"] Oct 30 19:40:09.682579 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:09.682563 4531 cpu_manager.go:414] "containerMap: removing container" podUID="ba782ca9-28c4-4df0-b467-b727d2b1c70b" containerName="revoke-keys" Oct 30 19:40:09.682632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:09.682580 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba782ca9-28c4-4df0-b467-b727d2b1c70b" containerName="revoke-keys" Oct 30 19:40:09.682692 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:09.682682 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba782ca9-28c4-4df0-b467-b727d2b1c70b" containerName="revoke-keys" Oct 30 19:40:09.686073 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:09.686048 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2" Oct 30 19:40:09.759419 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:09.759369 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a5c809-3bd4-4915-ab4b-3822a946ea56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2\" (UID: \"43a5c809-3bd4-4915-ab4b-3822a946ea56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2" Oct 30 19:40:09.759617 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:09.759526 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2rn5\" (UniqueName: \"kubernetes.io/projected/43a5c809-3bd4-4915-ab4b-3822a946ea56-kube-api-access-m2rn5\") pod \"maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2\" (UID: \"43a5c809-3bd4-4915-ab4b-3822a946ea56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2" Oct 30 19:40:09.759617 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:09.759605 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a5c809-3bd4-4915-ab4b-3822a946ea56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2\" (UID: \"43a5c809-3bd4-4915-ab4b-3822a946ea56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2" Oct 30 19:40:09.788345 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:09.788312 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:40:09.788345 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:09.788331 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:40:09.788587 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:09.788366 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:40:09.788587 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:09.788421 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:40:09.791037 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:09.790996 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2"] Oct 30 19:40:09.860674 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:09.860629 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a5c809-3bd4-4915-ab4b-3822a946ea56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2\" (UID: \"43a5c809-3bd4-4915-ab4b-3822a946ea56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2" Oct 30 19:40:09.860909 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:09.860726 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2rn5\" (UniqueName: \"kubernetes.io/projected/43a5c809-3bd4-4915-ab4b-3822a946ea56-kube-api-access-m2rn5\") pod \"maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2\" (UID: \"43a5c809-3bd4-4915-ab4b-3822a946ea56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2" Oct 30 19:40:09.860909 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:09.860797 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a5c809-3bd4-4915-ab4b-3822a946ea56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2\" (UID: \"43a5c809-3bd4-4915-ab4b-3822a946ea56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2" Oct 30 19:40:09.861364 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:09.861342 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a5c809-3bd4-4915-ab4b-3822a946ea56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2\" (UID: \"43a5c809-3bd4-4915-ab4b-3822a946ea56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2" Oct 30 19:40:09.863510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:09.863481 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a5c809-3bd4-4915-ab4b-3822a946ea56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2\" (UID: \"43a5c809-3bd4-4915-ab4b-3822a946ea56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2" Oct 30 19:40:09.924810 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:09.924770 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2rn5\" (UniqueName: \"kubernetes.io/projected/43a5c809-3bd4-4915-ab4b-3822a946ea56-kube-api-access-m2rn5\") pod \"maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2\" (UID: \"43a5c809-3bd4-4915-ab4b-3822a946ea56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2" Oct 30 19:40:10.001140 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:10.001092 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2" Oct 30 19:40:10.386537 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:10.386509 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2"] Oct 30 19:40:10.388985 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:40:10.388942 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a5c809_3bd4_4915_ab4b_3822a946ea56.slice/crio-45aa222de148acee74340d6e35c8c5b72ba28976d7e78b328ed5db2ecbad1d82 WatchSource:0}: Error finding container 45aa222de148acee74340d6e35c8c5b72ba28976d7e78b328ed5db2ecbad1d82: Status 404 returned error can't find the container with id 45aa222de148acee74340d6e35c8c5b72ba28976d7e78b328ed5db2ecbad1d82 Oct 30 19:40:10.745521 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:10.745458 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2" event={"ID":"43a5c809-3bd4-4915-ab4b-3822a946ea56","Type":"ContainerStarted","Data":"e3a091e167ed169c13ade784e43a33b4ba4d7a246fd35d0035f29e87e0405650"} Oct 30 19:40:10.745983 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:10.745540 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2" event={"ID":"43a5c809-3bd4-4915-ab4b-3822a946ea56","Type":"ContainerStarted","Data":"45aa222de148acee74340d6e35c8c5b72ba28976d7e78b328ed5db2ecbad1d82"} Oct 30 19:40:10.862410 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:10.862340 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2" podStartSLOduration=1.862317346 podStartE2EDuration="1.862317346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:10.861156511 +0000 UTC m=+3970.888261470" watchObservedRunningTime="2026-10-30 19:40:10.862317346 +0000 UTC m=+3970.889422308" Oct 30 19:40:11.753036 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:11.753003 4531 generic.go:356] "Generic (PLEG): container finished" podID="43a5c809-3bd4-4915-ab4b-3822a946ea56" containerID="e3a091e167ed169c13ade784e43a33b4ba4d7a246fd35d0035f29e87e0405650" exitCode=0 Oct 30 19:40:11.753518 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:11.753088 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2" event={"ID":"43a5c809-3bd4-4915-ab4b-3822a946ea56","Type":"ContainerDied","Data":"e3a091e167ed169c13ade784e43a33b4ba4d7a246fd35d0035f29e87e0405650"} Oct 30 19:40:12.918761 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:12.918727 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2" Oct 30 19:40:13.031735 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:13.031630 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4"] Oct 30 19:40:13.036079 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:13.036040 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3b6013-slxq4"] Oct 30 19:40:13.107478 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:13.107437 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a5c809-3bd4-4915-ab4b-3822a946ea56-kube-api-access-m2rn5\" (UniqueName: \"kubernetes.io/projected/43a5c809-3bd4-4915-ab4b-3822a946ea56-kube-api-access-m2rn5\") pod \"43a5c809-3bd4-4915-ab4b-3822a946ea56\" (UID: \"43a5c809-3bd4-4915-ab4b-3822a946ea56\") " Oct 30 19:40:13.107681 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:13.107596 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43a5c809-3bd4-4915-ab4b-3822a946ea56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a5c809-3bd4-4915-ab4b-3822a946ea56-maas-api-xks-ca\") pod \"43a5c809-3bd4-4915-ab4b-3822a946ea56\" (UID: \"43a5c809-3bd4-4915-ab4b-3822a946ea56\") " Oct 30 19:40:13.107681 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:13.107622 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43a5c809-3bd4-4915-ab4b-3822a946ea56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a5c809-3bd4-4915-ab4b-3822a946ea56-maas-api-openshift-ca\") pod \"43a5c809-3bd4-4915-ab4b-3822a946ea56\" (UID: \"43a5c809-3bd4-4915-ab4b-3822a946ea56\") " Oct 30 19:40:13.108036 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:13.108009 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43a5c809-3bd4-4915-ab4b-3822a946ea56-maas-api-openshift-ca" pod "43a5c809-3bd4-4915-ab4b-3822a946ea56" (UID: "43a5c809-3bd4-4915-ab4b-3822a946ea56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:13.109850 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:13.109817 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a5c809-3bd4-4915-ab4b-3822a946ea56-kube-api-access-m2rn5" pod "43a5c809-3bd4-4915-ab4b-3822a946ea56" (UID: "43a5c809-3bd4-4915-ab4b-3822a946ea56"). InnerVolumeSpecName "kube-api-access-m2rn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:13.109982 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:13.109877 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43a5c809-3bd4-4915-ab4b-3822a946ea56-maas-api-xks-ca" pod "43a5c809-3bd4-4915-ab4b-3822a946ea56" (UID: "43a5c809-3bd4-4915-ab4b-3822a946ea56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:40:13.209062 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:13.209016 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2rn5\" (UniqueName: \"kubernetes.io/projected/43a5c809-3bd4-4915-ab4b-3822a946ea56-kube-api-access-m2rn5\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:40:13.209062 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:13.209055 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a5c809-3bd4-4915-ab4b-3822a946ea56-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:40:13.209062 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:13.209067 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a5c809-3bd4-4915-ab4b-3822a946ea56-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:40:13.767975 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:13.767943 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2" Oct 30 19:40:13.768191 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:13.767943 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2" event={"ID":"43a5c809-3bd4-4915-ab4b-3822a946ea56","Type":"ContainerDied","Data":"45aa222de148acee74340d6e35c8c5b72ba28976d7e78b328ed5db2ecbad1d82"} Oct 30 19:40:13.768191 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:13.768069 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45aa222de148acee74340d6e35c8c5b72ba28976d7e78b328ed5db2ecbad1d82" Oct 30 19:40:14.098164 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:14.098069 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1317539-30ac-4327-bf57-2164195f6a44" path="/var/lib/kubelet/pods/c1317539-30ac-4327-bf57-2164195f6a44/volumes" Oct 30 19:40:19.255698 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:19.255661 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd"] Oct 30 19:40:19.305010 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:19.304965 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-25e5ef-wzwmd"] Oct 30 19:40:20.100608 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:20.100567 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c283bdf-4f79-45c2-bef8-b95f716d2aff" path="/var/lib/kubelet/pods/5c283bdf-4f79-45c2-bef8-b95f716d2aff/volumes" Oct 30 19:40:21.194483 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:21.194438 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g"] Oct 30 19:40:26.195447 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:26.195386 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g" podUID="e9b41342-7ece-47b2-bb04-129a141a0a2d" containerName="payload-processing" containerID="cri-o://2ba0ef5b35515278e9731068a43b0913263293c6ccf5b900a2eec38cd2d5dfe5" gracePeriod=25 Oct 30 19:40:26.419082 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:40:26.417697 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:40:26.482219 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:26.477882 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g" Oct 30 19:40:26.559789 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:26.559734 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9b41342-7ece-47b2-bb04-129a141a0a2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9b41342-7ece-47b2-bb04-129a141a0a2d-plugins-config-volume\") pod \"e9b41342-7ece-47b2-bb04-129a141a0a2d\" (UID: \"e9b41342-7ece-47b2-bb04-129a141a0a2d\") " Oct 30 19:40:26.559789 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:26.559791 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9b41342-7ece-47b2-bb04-129a141a0a2d-kube-api-access-mx5dn\" (UniqueName: \"kubernetes.io/projected/e9b41342-7ece-47b2-bb04-129a141a0a2d-kube-api-access-mx5dn\") pod \"e9b41342-7ece-47b2-bb04-129a141a0a2d\" (UID: \"e9b41342-7ece-47b2-bb04-129a141a0a2d\") " Oct 30 19:40:26.560270 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:26.560242 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9b41342-7ece-47b2-bb04-129a141a0a2d-plugins-config-volume" pod "e9b41342-7ece-47b2-bb04-129a141a0a2d" (UID: "e9b41342-7ece-47b2-bb04-129a141a0a2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:26.562420 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:26.562381 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9b41342-7ece-47b2-bb04-129a141a0a2d-kube-api-access-mx5dn" pod "e9b41342-7ece-47b2-bb04-129a141a0a2d" (UID: "e9b41342-7ece-47b2-bb04-129a141a0a2d"). InnerVolumeSpecName "kube-api-access-mx5dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:26.664901 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:26.661504 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9b41342-7ece-47b2-bb04-129a141a0a2d-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:40:26.664901 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:26.661548 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx5dn\" (UniqueName: \"kubernetes.io/projected/e9b41342-7ece-47b2-bb04-129a141a0a2d-kube-api-access-mx5dn\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:40:26.850659 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:26.850619 4531 generic.go:356] "Generic (PLEG): container finished" podID="e9b41342-7ece-47b2-bb04-129a141a0a2d" containerID="2ba0ef5b35515278e9731068a43b0913263293c6ccf5b900a2eec38cd2d5dfe5" exitCode=0 Oct 30 19:40:26.850868 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:26.850670 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g" Oct 30 19:40:26.850868 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:26.850714 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g" event={"ID":"e9b41342-7ece-47b2-bb04-129a141a0a2d","Type":"ContainerDied","Data":"2ba0ef5b35515278e9731068a43b0913263293c6ccf5b900a2eec38cd2d5dfe5"} Oct 30 19:40:26.850868 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:26.850779 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g" event={"ID":"e9b41342-7ece-47b2-bb04-129a141a0a2d","Type":"ContainerDied","Data":"d99a55507018aeb8a857bf277ba9b1b38623ee1453e4e60ca572f94dd778cc74"} Oct 30 19:40:26.850868 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:26.850815 4531 scope.go:122] "RemoveContainer" containerID="2ba0ef5b35515278e9731068a43b0913263293c6ccf5b900a2eec38cd2d5dfe5" Oct 30 19:40:26.861950 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:26.861855 4531 scope.go:122] "RemoveContainer" containerID="2ba0ef5b35515278e9731068a43b0913263293c6ccf5b900a2eec38cd2d5dfe5" Oct 30 19:40:26.862337 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:40:26.862301 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ba0ef5b35515278e9731068a43b0913263293c6ccf5b900a2eec38cd2d5dfe5\": container with ID starting with 2ba0ef5b35515278e9731068a43b0913263293c6ccf5b900a2eec38cd2d5dfe5 not found: ID does not exist" containerID="2ba0ef5b35515278e9731068a43b0913263293c6ccf5b900a2eec38cd2d5dfe5" Oct 30 19:40:26.862449 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:26.862348 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ba0ef5b35515278e9731068a43b0913263293c6ccf5b900a2eec38cd2d5dfe5"} err="failed to get container status \"2ba0ef5b35515278e9731068a43b0913263293c6ccf5b900a2eec38cd2d5dfe5\": rpc error: code = NotFound desc = could not find container \"2ba0ef5b35515278e9731068a43b0913263293c6ccf5b900a2eec38cd2d5dfe5\": container with ID starting with 2ba0ef5b35515278e9731068a43b0913263293c6ccf5b900a2eec38cd2d5dfe5 not found: ID does not exist" Oct 30 19:40:27.015584 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:27.015250 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g"] Oct 30 19:40:27.017846 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:27.017803 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-345670-644777d7f6-pfc8g"] Oct 30 19:40:27.418412 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:27.418376 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb"] Oct 30 19:40:27.418844 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:27.418828 4531 cpu_manager.go:414] "containerMap: removing container" podUID="43a5c809-3bd4-4915-ab4b-3822a946ea56" containerName="revoke-keys" Oct 30 19:40:27.418889 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:27.418846 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a5c809-3bd4-4915-ab4b-3822a946ea56" containerName="revoke-keys" Oct 30 19:40:27.418889 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:27.418857 4531 cpu_manager.go:414] "containerMap: removing container" podUID="e9b41342-7ece-47b2-bb04-129a141a0a2d" containerName="payload-processing" Oct 30 19:40:27.418889 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:27.418862 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9b41342-7ece-47b2-bb04-129a141a0a2d" containerName="payload-processing" Oct 30 19:40:27.418991 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:27.418934 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a5c809-3bd4-4915-ab4b-3822a946ea56" containerName="revoke-keys" Oct 30 19:40:27.418991 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:27.418944 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9b41342-7ece-47b2-bb04-129a141a0a2d" containerName="payload-processing" Oct 30 19:40:27.423234 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:27.423213 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb" Oct 30 19:40:27.469372 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:27.469331 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7649797-82f3-4199-9d40-d2e802a3eac9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb\" (UID: \"a7649797-82f3-4199-9d40-d2e802a3eac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb" Oct 30 19:40:27.469372 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:27.469368 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfm7f\" (UniqueName: \"kubernetes.io/projected/a7649797-82f3-4199-9d40-d2e802a3eac9-kube-api-access-jfm7f\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb\" (UID: \"a7649797-82f3-4199-9d40-d2e802a3eac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb" Oct 30 19:40:27.469618 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:27.469555 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7649797-82f3-4199-9d40-d2e802a3eac9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb\" (UID: \"a7649797-82f3-4199-9d40-d2e802a3eac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb" Oct 30 19:40:27.481446 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:27.481415 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb"] Oct 30 19:40:27.481730 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:27.481710 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:40:27.481811 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:27.481734 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:40:27.481864 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:27.481828 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:40:27.481864 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:27.481831 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:40:27.570564 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:27.570514 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7649797-82f3-4199-9d40-d2e802a3eac9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb\" (UID: \"a7649797-82f3-4199-9d40-d2e802a3eac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb" Oct 30 19:40:27.570790 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:27.570663 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7649797-82f3-4199-9d40-d2e802a3eac9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb\" (UID: \"a7649797-82f3-4199-9d40-d2e802a3eac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb" Oct 30 19:40:27.570790 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:27.570688 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfm7f\" (UniqueName: \"kubernetes.io/projected/a7649797-82f3-4199-9d40-d2e802a3eac9-kube-api-access-jfm7f\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb\" (UID: \"a7649797-82f3-4199-9d40-d2e802a3eac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb" Oct 30 19:40:27.571450 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:27.571427 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7649797-82f3-4199-9d40-d2e802a3eac9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb\" (UID: \"a7649797-82f3-4199-9d40-d2e802a3eac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb" Oct 30 19:40:27.573370 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:27.573344 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7649797-82f3-4199-9d40-d2e802a3eac9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb\" (UID: \"a7649797-82f3-4199-9d40-d2e802a3eac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb" Oct 30 19:40:27.632753 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:27.632715 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfm7f\" (UniqueName: \"kubernetes.io/projected/a7649797-82f3-4199-9d40-d2e802a3eac9-kube-api-access-jfm7f\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb\" (UID: \"a7649797-82f3-4199-9d40-d2e802a3eac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb" Oct 30 19:40:27.736694 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:27.736590 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb" Oct 30 19:40:28.098628 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:28.098588 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9b41342-7ece-47b2-bb04-129a141a0a2d" path="/var/lib/kubelet/pods/e9b41342-7ece-47b2-bb04-129a141a0a2d/volumes" Oct 30 19:40:28.119807 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:28.119775 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb"] Oct 30 19:40:28.122087 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:40:28.122052 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7649797_82f3_4199_9d40_d2e802a3eac9.slice/crio-d1f70f5014ac15cc0f4fcd554fd8a404edfa90b6eef74b46865794546b875eab WatchSource:0}: Error finding container d1f70f5014ac15cc0f4fcd554fd8a404edfa90b6eef74b46865794546b875eab: Status 404 returned error can't find the container with id d1f70f5014ac15cc0f4fcd554fd8a404edfa90b6eef74b46865794546b875eab Oct 30 19:40:28.879526 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:28.879486 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb" event={"ID":"a7649797-82f3-4199-9d40-d2e802a3eac9","Type":"ContainerStarted","Data":"92bc7b29e8281de1e85ad60206da4e2a6360b3b8d3f9f01079d8079c65bf564d"} Oct 30 19:40:28.879992 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:28.879544 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb" event={"ID":"a7649797-82f3-4199-9d40-d2e802a3eac9","Type":"ContainerStarted","Data":"d1f70f5014ac15cc0f4fcd554fd8a404edfa90b6eef74b46865794546b875eab"} Oct 30 19:40:28.997146 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:28.997066 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb" podStartSLOduration=1.997047299 podStartE2EDuration="1.997047299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:28.99498194 +0000 UTC m=+3989.022086901" watchObservedRunningTime="2026-10-30 19:40:28.997047299 +0000 UTC m=+3989.024152287" Oct 30 19:40:29.887482 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:29.887437 4531 generic.go:356] "Generic (PLEG): container finished" podID="a7649797-82f3-4199-9d40-d2e802a3eac9" containerID="92bc7b29e8281de1e85ad60206da4e2a6360b3b8d3f9f01079d8079c65bf564d" exitCode=0 Oct 30 19:40:29.887953 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:29.887509 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb" event={"ID":"a7649797-82f3-4199-9d40-d2e802a3eac9","Type":"ContainerDied","Data":"92bc7b29e8281de1e85ad60206da4e2a6360b3b8d3f9f01079d8079c65bf564d"} Oct 30 19:40:31.037438 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:31.037408 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb" Oct 30 19:40:31.112624 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:31.112577 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7649797-82f3-4199-9d40-d2e802a3eac9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7649797-82f3-4199-9d40-d2e802a3eac9-maas-api-openshift-ca\") pod \"a7649797-82f3-4199-9d40-d2e802a3eac9\" (UID: \"a7649797-82f3-4199-9d40-d2e802a3eac9\") " Oct 30 19:40:31.112624 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:31.112616 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7649797-82f3-4199-9d40-d2e802a3eac9-kube-api-access-jfm7f\" (UniqueName: \"kubernetes.io/projected/a7649797-82f3-4199-9d40-d2e802a3eac9-kube-api-access-jfm7f\") pod \"a7649797-82f3-4199-9d40-d2e802a3eac9\" (UID: \"a7649797-82f3-4199-9d40-d2e802a3eac9\") " Oct 30 19:40:31.112887 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:31.112679 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7649797-82f3-4199-9d40-d2e802a3eac9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7649797-82f3-4199-9d40-d2e802a3eac9-maas-api-xks-ca\") pod \"a7649797-82f3-4199-9d40-d2e802a3eac9\" (UID: \"a7649797-82f3-4199-9d40-d2e802a3eac9\") " Oct 30 19:40:31.113074 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:31.113045 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7649797-82f3-4199-9d40-d2e802a3eac9-maas-api-openshift-ca" pod "a7649797-82f3-4199-9d40-d2e802a3eac9" (UID: "a7649797-82f3-4199-9d40-d2e802a3eac9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:31.115143 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:31.115099 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7649797-82f3-4199-9d40-d2e802a3eac9-maas-api-xks-ca" pod "a7649797-82f3-4199-9d40-d2e802a3eac9" (UID: "a7649797-82f3-4199-9d40-d2e802a3eac9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:40:31.115237 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:31.115172 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7649797-82f3-4199-9d40-d2e802a3eac9-kube-api-access-jfm7f" pod "a7649797-82f3-4199-9d40-d2e802a3eac9" (UID: "a7649797-82f3-4199-9d40-d2e802a3eac9"). InnerVolumeSpecName "kube-api-access-jfm7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:31.214662 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:31.214552 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7649797-82f3-4199-9d40-d2e802a3eac9-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:40:31.214662 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:31.214598 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfm7f\" (UniqueName: \"kubernetes.io/projected/a7649797-82f3-4199-9d40-d2e802a3eac9-kube-api-access-jfm7f\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:40:31.214662 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:31.214613 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7649797-82f3-4199-9d40-d2e802a3eac9-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:40:31.901935 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:31.901900 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb" Oct 30 19:40:31.902162 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:31.901936 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb" event={"ID":"a7649797-82f3-4199-9d40-d2e802a3eac9","Type":"ContainerDied","Data":"d1f70f5014ac15cc0f4fcd554fd8a404edfa90b6eef74b46865794546b875eab"} Oct 30 19:40:31.902162 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:31.901989 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1f70f5014ac15cc0f4fcd554fd8a404edfa90b6eef74b46865794546b875eab" Oct 30 19:40:36.518069 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:36.518014 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc"] Oct 30 19:40:36.567236 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:36.567197 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8f9c6b-4fshc"] Oct 30 19:40:38.098864 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:38.098823 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba782ca9-28c4-4df0-b467-b727d2b1c70b" path="/var/lib/kubelet/pods/ba782ca9-28c4-4df0-b467-b727d2b1c70b/volumes" Oct 30 19:40:45.124938 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:45.124836 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p"] Oct 30 19:40:45.125578 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:45.125556 4531 cpu_manager.go:414] "containerMap: removing container" podUID="a7649797-82f3-4199-9d40-d2e802a3eac9" containerName="revoke-keys" Oct 30 19:40:45.125700 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:45.125580 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7649797-82f3-4199-9d40-d2e802a3eac9" containerName="revoke-keys" Oct 30 19:40:45.125700 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:45.125695 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7649797-82f3-4199-9d40-d2e802a3eac9" containerName="revoke-keys" Oct 30 19:40:45.129287 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:45.129254 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p" Oct 30 19:40:45.179689 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:45.179647 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gjxl\" (UniqueName: \"kubernetes.io/projected/f29b3c44-0710-41cd-983f-553d775f2593-kube-api-access-5gjxl\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p\" (UID: \"f29b3c44-0710-41cd-983f-553d775f2593\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p" Oct 30 19:40:45.179867 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:45.179813 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f29b3c44-0710-41cd-983f-553d775f2593-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p\" (UID: \"f29b3c44-0710-41cd-983f-553d775f2593\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p" Oct 30 19:40:45.179867 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:45.179858 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f29b3c44-0710-41cd-983f-553d775f2593-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p\" (UID: \"f29b3c44-0710-41cd-983f-553d775f2593\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p" Oct 30 19:40:45.186715 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:45.186683 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:40:45.236273 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:45.236225 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:40:45.236273 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:45.236230 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:40:45.236530 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:45.236395 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:40:45.238989 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:45.238960 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p"] Oct 30 19:40:45.280864 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:45.280821 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gjxl\" (UniqueName: \"kubernetes.io/projected/f29b3c44-0710-41cd-983f-553d775f2593-kube-api-access-5gjxl\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p\" (UID: \"f29b3c44-0710-41cd-983f-553d775f2593\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p" Oct 30 19:40:45.281053 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:45.280930 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f29b3c44-0710-41cd-983f-553d775f2593-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p\" (UID: \"f29b3c44-0710-41cd-983f-553d775f2593\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p" Oct 30 19:40:45.281053 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:45.280966 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f29b3c44-0710-41cd-983f-553d775f2593-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p\" (UID: \"f29b3c44-0710-41cd-983f-553d775f2593\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p" Oct 30 19:40:45.281773 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:45.281704 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f29b3c44-0710-41cd-983f-553d775f2593-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p\" (UID: \"f29b3c44-0710-41cd-983f-553d775f2593\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p" Oct 30 19:40:45.283687 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:45.283662 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f29b3c44-0710-41cd-983f-553d775f2593-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p\" (UID: \"f29b3c44-0710-41cd-983f-553d775f2593\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p" Oct 30 19:40:45.342633 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:45.342597 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gjxl\" (UniqueName: \"kubernetes.io/projected/f29b3c44-0710-41cd-983f-553d775f2593-kube-api-access-5gjxl\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p\" (UID: \"f29b3c44-0710-41cd-983f-553d775f2593\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p" Oct 30 19:40:45.443631 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:45.443534 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p" Oct 30 19:40:45.833805 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:45.833776 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p"] Oct 30 19:40:45.835576 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:40:45.835543 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29b3c44_0710_41cd_983f_553d775f2593.slice/crio-b9ad9c617b01bde912ef6ca74e73eedff4826f6465d24d041271ca27128460e8 WatchSource:0}: Error finding container b9ad9c617b01bde912ef6ca74e73eedff4826f6465d24d041271ca27128460e8: Status 404 returned error can't find the container with id b9ad9c617b01bde912ef6ca74e73eedff4826f6465d24d041271ca27128460e8 Oct 30 19:40:45.988201 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:45.988157 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p" event={"ID":"f29b3c44-0710-41cd-983f-553d775f2593","Type":"ContainerStarted","Data":"b9ad9c617b01bde912ef6ca74e73eedff4826f6465d24d041271ca27128460e8"} Oct 30 19:40:46.996364 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:46.996330 4531 generic.go:356] "Generic (PLEG): container finished" podID="f29b3c44-0710-41cd-983f-553d775f2593" containerID="1e5140215ae2f72a7c94a2d576697ccd961bf53878035cf9e4139404a16b955f" exitCode=0 Oct 30 19:40:46.996791 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:46.996418 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p" event={"ID":"f29b3c44-0710-41cd-983f-553d775f2593","Type":"ContainerDied","Data":"1e5140215ae2f72a7c94a2d576697ccd961bf53878035cf9e4139404a16b955f"} Oct 30 19:40:48.164503 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:48.164468 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p" Oct 30 19:40:48.212009 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:48.211965 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f29b3c44-0710-41cd-983f-553d775f2593-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f29b3c44-0710-41cd-983f-553d775f2593-maas-api-openshift-ca\") pod \"f29b3c44-0710-41cd-983f-553d775f2593\" (UID: \"f29b3c44-0710-41cd-983f-553d775f2593\") " Oct 30 19:40:48.212267 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:48.212082 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f29b3c44-0710-41cd-983f-553d775f2593-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f29b3c44-0710-41cd-983f-553d775f2593-maas-api-xks-ca\") pod \"f29b3c44-0710-41cd-983f-553d775f2593\" (UID: \"f29b3c44-0710-41cd-983f-553d775f2593\") " Oct 30 19:40:48.212267 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:48.212181 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f29b3c44-0710-41cd-983f-553d775f2593-kube-api-access-5gjxl\" (UniqueName: \"kubernetes.io/projected/f29b3c44-0710-41cd-983f-553d775f2593-kube-api-access-5gjxl\") pod \"f29b3c44-0710-41cd-983f-553d775f2593\" (UID: \"f29b3c44-0710-41cd-983f-553d775f2593\") " Oct 30 19:40:48.212484 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:48.212458 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f29b3c44-0710-41cd-983f-553d775f2593-maas-api-openshift-ca" pod "f29b3c44-0710-41cd-983f-553d775f2593" (UID: "f29b3c44-0710-41cd-983f-553d775f2593"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:48.212592 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:48.212569 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f29b3c44-0710-41cd-983f-553d775f2593-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:40:48.214615 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:48.214575 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f29b3c44-0710-41cd-983f-553d775f2593-maas-api-xks-ca" pod "f29b3c44-0710-41cd-983f-553d775f2593" (UID: "f29b3c44-0710-41cd-983f-553d775f2593"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:40:48.214615 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:48.214603 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f29b3c44-0710-41cd-983f-553d775f2593-kube-api-access-5gjxl" pod "f29b3c44-0710-41cd-983f-553d775f2593" (UID: "f29b3c44-0710-41cd-983f-553d775f2593"). InnerVolumeSpecName "kube-api-access-5gjxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:48.313516 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:48.313394 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f29b3c44-0710-41cd-983f-553d775f2593-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:40:48.313516 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:48.313445 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gjxl\" (UniqueName: \"kubernetes.io/projected/f29b3c44-0710-41cd-983f-553d775f2593-kube-api-access-5gjxl\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:40:49.010378 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:49.010343 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p" Oct 30 19:40:49.010579 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:49.010373 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p" event={"ID":"f29b3c44-0710-41cd-983f-553d775f2593","Type":"ContainerDied","Data":"b9ad9c617b01bde912ef6ca74e73eedff4826f6465d24d041271ca27128460e8"} Oct 30 19:40:49.010579 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:49.010450 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9ad9c617b01bde912ef6ca74e73eedff4826f6465d24d041271ca27128460e8" Oct 30 19:40:56.400356 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:40:56.400314 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:40:56.653459 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:56.653363 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr"] Oct 30 19:40:56.757665 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:40:56.757628 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s"] Oct 30 19:41:01.654602 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:01.654548 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr" podUID="eb431a4c-7eba-4bf5-ae56-56cba97e5342" containerName="payload-processing" containerID="cri-o://7f2c676dd8a5494522d146d5888eb58c40baddfb46fee52feb28ccb1bee42b06" gracePeriod=25 Oct 30 19:41:01.758222 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:01.758106 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s" podUID="cc0b354d-95e7-477f-b479-2028dcc5a950" containerName="payload-pre-processing" containerID="cri-o://a0510b235e5bc0cc1b090359f69c32afbaa15e1bff8a79cfeb48b09cc2840303" gracePeriod=25 Oct 30 19:41:02.907578 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:02.907542 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s" podUID="cc0b354d-95e7-477f-b479-2028dcc5a950" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:41:02.907990 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:02.907538 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr" podUID="eb431a4c-7eba-4bf5-ae56-56cba97e5342" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:41:12.446705 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:12.446639 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2"] Oct 30 19:41:12.495944 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:12.495900 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-345670-5jhw2"] Oct 30 19:41:12.907091 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:12.907048 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s" podUID="cc0b354d-95e7-477f-b479-2028dcc5a950" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:41:12.907315 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:12.907053 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr" podUID="eb431a4c-7eba-4bf5-ae56-56cba97e5342" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:41:14.098304 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:14.098263 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43a5c809-3bd4-4915-ab4b-3822a946ea56" path="/var/lib/kubelet/pods/43a5c809-3bd4-4915-ab4b-3822a946ea56/volumes" Oct 30 19:41:20.814671 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:20.814628 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph"] Oct 30 19:41:20.815166 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:20.815078 4531 cpu_manager.go:414] "containerMap: removing container" podUID="f29b3c44-0710-41cd-983f-553d775f2593" containerName="revoke-keys" Oct 30 19:41:20.815166 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:20.815090 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f29b3c44-0710-41cd-983f-553d775f2593" containerName="revoke-keys" Oct 30 19:41:20.815269 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:20.815206 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f29b3c44-0710-41cd-983f-553d775f2593" containerName="revoke-keys" Oct 30 19:41:20.818791 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:20.818766 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph" Oct 30 19:41:20.870777 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:20.870737 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:41:20.871307 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:20.871289 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:41:20.875117 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:20.875075 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:41:20.878793 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:20.878762 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a975b396-b01f-42e8-a238-2b363ad846cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph\" (UID: \"a975b396-b01f-42e8-a238-2b363ad846cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph" Oct 30 19:41:20.879039 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:20.879014 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b9hf\" (UniqueName: \"kubernetes.io/projected/a975b396-b01f-42e8-a238-2b363ad846cb-kube-api-access-7b9hf\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph\" (UID: \"a975b396-b01f-42e8-a238-2b363ad846cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph" Oct 30 19:41:20.879095 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:20.879058 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a975b396-b01f-42e8-a238-2b363ad846cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph\" (UID: \"a975b396-b01f-42e8-a238-2b363ad846cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph" Oct 30 19:41:20.922839 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:20.922799 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph"] Oct 30 19:41:20.974921 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:20.974881 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:41:20.980528 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:20.980493 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a975b396-b01f-42e8-a238-2b363ad846cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph\" (UID: \"a975b396-b01f-42e8-a238-2b363ad846cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph" Oct 30 19:41:20.980652 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:20.980639 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b9hf\" (UniqueName: \"kubernetes.io/projected/a975b396-b01f-42e8-a238-2b363ad846cb-kube-api-access-7b9hf\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph\" (UID: \"a975b396-b01f-42e8-a238-2b363ad846cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph" Oct 30 19:41:20.980698 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:20.980669 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a975b396-b01f-42e8-a238-2b363ad846cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph\" (UID: \"a975b396-b01f-42e8-a238-2b363ad846cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph" Oct 30 19:41:20.981165 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:20.981123 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a975b396-b01f-42e8-a238-2b363ad846cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph\" (UID: \"a975b396-b01f-42e8-a238-2b363ad846cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph" Oct 30 19:41:20.983528 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:20.983498 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a975b396-b01f-42e8-a238-2b363ad846cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph\" (UID: \"a975b396-b01f-42e8-a238-2b363ad846cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph" Oct 30 19:41:21.044265 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:21.044233 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b9hf\" (UniqueName: \"kubernetes.io/projected/a975b396-b01f-42e8-a238-2b363ad846cb-kube-api-access-7b9hf\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph\" (UID: \"a975b396-b01f-42e8-a238-2b363ad846cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph" Oct 30 19:41:21.134032 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:21.133916 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph" Oct 30 19:41:21.523466 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:21.523432 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph"] Oct 30 19:41:21.525846 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:41:21.525815 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda975b396_b01f_42e8_a238_2b363ad846cb.slice/crio-0339edd0654cc5c8ef55397df1a57c8adf05e222cf4af4b6155b3631ca7e4507 WatchSource:0}: Error finding container 0339edd0654cc5c8ef55397df1a57c8adf05e222cf4af4b6155b3631ca7e4507: Status 404 returned error can't find the container with id 0339edd0654cc5c8ef55397df1a57c8adf05e222cf4af4b6155b3631ca7e4507 Oct 30 19:41:21.911242 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:21.911204 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr" Oct 30 19:41:21.993277 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:21.993248 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb431a4c-7eba-4bf5-ae56-56cba97e5342-kube-api-access-gtllg\" (UniqueName: \"kubernetes.io/projected/eb431a4c-7eba-4bf5-ae56-56cba97e5342-kube-api-access-gtllg\") pod \"eb431a4c-7eba-4bf5-ae56-56cba97e5342\" (UID: \"eb431a4c-7eba-4bf5-ae56-56cba97e5342\") " Oct 30 19:41:21.993545 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:21.993336 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb431a4c-7eba-4bf5-ae56-56cba97e5342-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb431a4c-7eba-4bf5-ae56-56cba97e5342-plugins-config-volume\") pod \"eb431a4c-7eba-4bf5-ae56-56cba97e5342\" (UID: \"eb431a4c-7eba-4bf5-ae56-56cba97e5342\") " Oct 30 19:41:21.994122 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:21.994090 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb431a4c-7eba-4bf5-ae56-56cba97e5342-plugins-config-volume" pod "eb431a4c-7eba-4bf5-ae56-56cba97e5342" (UID: "eb431a4c-7eba-4bf5-ae56-56cba97e5342"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:21.995806 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:21.995770 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb431a4c-7eba-4bf5-ae56-56cba97e5342-kube-api-access-gtllg" pod "eb431a4c-7eba-4bf5-ae56-56cba97e5342" (UID: "eb431a4c-7eba-4bf5-ae56-56cba97e5342"). InnerVolumeSpecName "kube-api-access-gtllg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:22.025145 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.025106 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s" Oct 30 19:41:22.094608 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.094501 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc0b354d-95e7-477f-b479-2028dcc5a950-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0b354d-95e7-477f-b479-2028dcc5a950-plugins-config-volume\") pod \"cc0b354d-95e7-477f-b479-2028dcc5a950\" (UID: \"cc0b354d-95e7-477f-b479-2028dcc5a950\") " Oct 30 19:41:22.094788 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.094606 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc0b354d-95e7-477f-b479-2028dcc5a950-kube-api-access-fxrwp\" (UniqueName: \"kubernetes.io/projected/cc0b354d-95e7-477f-b479-2028dcc5a950-kube-api-access-fxrwp\") pod \"cc0b354d-95e7-477f-b479-2028dcc5a950\" (UID: \"cc0b354d-95e7-477f-b479-2028dcc5a950\") " Oct 30 19:41:22.094908 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.094893 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb431a4c-7eba-4bf5-ae56-56cba97e5342-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:41:22.094985 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.094908 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtllg\" (UniqueName: \"kubernetes.io/projected/eb431a4c-7eba-4bf5-ae56-56cba97e5342-kube-api-access-gtllg\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:41:22.094985 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.094957 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc0b354d-95e7-477f-b479-2028dcc5a950-plugins-config-volume" pod "cc0b354d-95e7-477f-b479-2028dcc5a950" (UID: "cc0b354d-95e7-477f-b479-2028dcc5a950"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:22.097462 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.097427 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0b354d-95e7-477f-b479-2028dcc5a950-kube-api-access-fxrwp" pod "cc0b354d-95e7-477f-b479-2028dcc5a950" (UID: "cc0b354d-95e7-477f-b479-2028dcc5a950"). InnerVolumeSpecName "kube-api-access-fxrwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:22.196798 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.196752 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0b354d-95e7-477f-b479-2028dcc5a950-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:41:22.196798 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.196798 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxrwp\" (UniqueName: \"kubernetes.io/projected/cc0b354d-95e7-477f-b479-2028dcc5a950-kube-api-access-fxrwp\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:41:22.224942 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.224897 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph" event={"ID":"a975b396-b01f-42e8-a238-2b363ad846cb","Type":"ContainerStarted","Data":"b55e7822b376114fdb82c9d7c55791fb7b01edfe75b8f5c6ea771e5e5ada4238"} Oct 30 19:41:22.225197 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.224970 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph" event={"ID":"a975b396-b01f-42e8-a238-2b363ad846cb","Type":"ContainerStarted","Data":"0339edd0654cc5c8ef55397df1a57c8adf05e222cf4af4b6155b3631ca7e4507"} Oct 30 19:41:22.226657 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.226629 4531 generic.go:356] "Generic (PLEG): container finished" podID="cc0b354d-95e7-477f-b479-2028dcc5a950" containerID="a0510b235e5bc0cc1b090359f69c32afbaa15e1bff8a79cfeb48b09cc2840303" exitCode=0 Oct 30 19:41:22.226811 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.226673 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s" Oct 30 19:41:22.226811 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.226715 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s" event={"ID":"cc0b354d-95e7-477f-b479-2028dcc5a950","Type":"ContainerDied","Data":"a0510b235e5bc0cc1b090359f69c32afbaa15e1bff8a79cfeb48b09cc2840303"} Oct 30 19:41:22.226811 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.226774 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s" event={"ID":"cc0b354d-95e7-477f-b479-2028dcc5a950","Type":"ContainerDied","Data":"f531a0d7514d0b0c693d40142352ef7217814fc6546f2b07c25b3e8da8755489"} Oct 30 19:41:22.226811 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.226802 4531 scope.go:122] "RemoveContainer" containerID="a0510b235e5bc0cc1b090359f69c32afbaa15e1bff8a79cfeb48b09cc2840303" Oct 30 19:41:22.228477 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.228448 4531 generic.go:356] "Generic (PLEG): container finished" podID="eb431a4c-7eba-4bf5-ae56-56cba97e5342" containerID="7f2c676dd8a5494522d146d5888eb58c40baddfb46fee52feb28ccb1bee42b06" exitCode=0 Oct 30 19:41:22.228607 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.228502 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr" Oct 30 19:41:22.228607 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.228556 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr" event={"ID":"eb431a4c-7eba-4bf5-ae56-56cba97e5342","Type":"ContainerDied","Data":"7f2c676dd8a5494522d146d5888eb58c40baddfb46fee52feb28ccb1bee42b06"} Oct 30 19:41:22.228724 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.228627 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr" event={"ID":"eb431a4c-7eba-4bf5-ae56-56cba97e5342","Type":"ContainerDied","Data":"f1b08390831e878cf8371ea68617119e7ce14fa67c34cd99aa2706395bd30630"} Oct 30 19:41:22.237744 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.237719 4531 scope.go:122] "RemoveContainer" containerID="a0510b235e5bc0cc1b090359f69c32afbaa15e1bff8a79cfeb48b09cc2840303" Oct 30 19:41:22.238179 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:41:22.238154 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0510b235e5bc0cc1b090359f69c32afbaa15e1bff8a79cfeb48b09cc2840303\": container with ID starting with a0510b235e5bc0cc1b090359f69c32afbaa15e1bff8a79cfeb48b09cc2840303 not found: ID does not exist" containerID="a0510b235e5bc0cc1b090359f69c32afbaa15e1bff8a79cfeb48b09cc2840303" Oct 30 19:41:22.238275 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.238192 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0510b235e5bc0cc1b090359f69c32afbaa15e1bff8a79cfeb48b09cc2840303"} err="failed to get container status \"a0510b235e5bc0cc1b090359f69c32afbaa15e1bff8a79cfeb48b09cc2840303\": rpc error: code = NotFound desc = could not find container \"a0510b235e5bc0cc1b090359f69c32afbaa15e1bff8a79cfeb48b09cc2840303\": container with ID starting with a0510b235e5bc0cc1b090359f69c32afbaa15e1bff8a79cfeb48b09cc2840303 not found: ID does not exist" Oct 30 19:41:22.238275 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.238212 4531 scope.go:122] "RemoveContainer" containerID="7f2c676dd8a5494522d146d5888eb58c40baddfb46fee52feb28ccb1bee42b06" Oct 30 19:41:22.248362 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.248336 4531 scope.go:122] "RemoveContainer" containerID="7f2c676dd8a5494522d146d5888eb58c40baddfb46fee52feb28ccb1bee42b06" Oct 30 19:41:22.248738 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:41:22.248707 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f2c676dd8a5494522d146d5888eb58c40baddfb46fee52feb28ccb1bee42b06\": container with ID starting with 7f2c676dd8a5494522d146d5888eb58c40baddfb46fee52feb28ccb1bee42b06 not found: ID does not exist" containerID="7f2c676dd8a5494522d146d5888eb58c40baddfb46fee52feb28ccb1bee42b06" Oct 30 19:41:22.248826 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.248751 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f2c676dd8a5494522d146d5888eb58c40baddfb46fee52feb28ccb1bee42b06"} err="failed to get container status \"7f2c676dd8a5494522d146d5888eb58c40baddfb46fee52feb28ccb1bee42b06\": rpc error: code = NotFound desc = could not find container \"7f2c676dd8a5494522d146d5888eb58c40baddfb46fee52feb28ccb1bee42b06\": container with ID starting with 7f2c676dd8a5494522d146d5888eb58c40baddfb46fee52feb28ccb1bee42b06 not found: ID does not exist" Oct 30 19:41:22.344786 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.344574 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph" podStartSLOduration=2.344553307 podStartE2EDuration="2.344553307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:22.340995018 +0000 UTC m=+4042.368099978" watchObservedRunningTime="2026-10-30 19:41:22.344553307 +0000 UTC m=+4042.371658267" Oct 30 19:41:22.452986 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.452943 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr"] Oct 30 19:41:22.458909 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.458859 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-dc5cdd-54475c499b-4lcnr"] Oct 30 19:41:22.566565 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.566523 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s"] Oct 30 19:41:22.573280 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:22.573235 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-dc5cdd-85664bcf47-s249s"] Oct 30 19:41:23.237962 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:23.237930 4531 generic.go:356] "Generic (PLEG): container finished" podID="a975b396-b01f-42e8-a238-2b363ad846cb" containerID="b55e7822b376114fdb82c9d7c55791fb7b01edfe75b8f5c6ea771e5e5ada4238" exitCode=0 Oct 30 19:41:23.238446 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:23.238027 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph" event={"ID":"a975b396-b01f-42e8-a238-2b363ad846cb","Type":"ContainerDied","Data":"b55e7822b376114fdb82c9d7c55791fb7b01edfe75b8f5c6ea771e5e5ada4238"} Oct 30 19:41:24.099608 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:24.099570 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc0b354d-95e7-477f-b479-2028dcc5a950" path="/var/lib/kubelet/pods/cc0b354d-95e7-477f-b479-2028dcc5a950/volumes" Oct 30 19:41:24.099989 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:24.099975 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb431a4c-7eba-4bf5-ae56-56cba97e5342" path="/var/lib/kubelet/pods/eb431a4c-7eba-4bf5-ae56-56cba97e5342/volumes" Oct 30 19:41:24.402562 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:24.402524 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph" Oct 30 19:41:24.523300 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:24.523264 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a975b396-b01f-42e8-a238-2b363ad846cb-kube-api-access-7b9hf\" (UniqueName: \"kubernetes.io/projected/a975b396-b01f-42e8-a238-2b363ad846cb-kube-api-access-7b9hf\") pod \"a975b396-b01f-42e8-a238-2b363ad846cb\" (UID: \"a975b396-b01f-42e8-a238-2b363ad846cb\") " Oct 30 19:41:24.523522 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:24.523315 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a975b396-b01f-42e8-a238-2b363ad846cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a975b396-b01f-42e8-a238-2b363ad846cb-maas-api-xks-ca\") pod \"a975b396-b01f-42e8-a238-2b363ad846cb\" (UID: \"a975b396-b01f-42e8-a238-2b363ad846cb\") " Oct 30 19:41:24.523522 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:24.523394 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a975b396-b01f-42e8-a238-2b363ad846cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a975b396-b01f-42e8-a238-2b363ad846cb-maas-api-openshift-ca\") pod \"a975b396-b01f-42e8-a238-2b363ad846cb\" (UID: \"a975b396-b01f-42e8-a238-2b363ad846cb\") " Oct 30 19:41:24.523991 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:24.523957 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a975b396-b01f-42e8-a238-2b363ad846cb-maas-api-openshift-ca" pod "a975b396-b01f-42e8-a238-2b363ad846cb" (UID: "a975b396-b01f-42e8-a238-2b363ad846cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:24.526473 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:24.526440 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a975b396-b01f-42e8-a238-2b363ad846cb-maas-api-xks-ca" pod "a975b396-b01f-42e8-a238-2b363ad846cb" (UID: "a975b396-b01f-42e8-a238-2b363ad846cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:41:24.526585 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:24.526476 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a975b396-b01f-42e8-a238-2b363ad846cb-kube-api-access-7b9hf" pod "a975b396-b01f-42e8-a238-2b363ad846cb" (UID: "a975b396-b01f-42e8-a238-2b363ad846cb"). InnerVolumeSpecName "kube-api-access-7b9hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:24.624777 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:24.624733 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a975b396-b01f-42e8-a238-2b363ad846cb-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:41:24.624777 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:24.624769 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b9hf\" (UniqueName: \"kubernetes.io/projected/a975b396-b01f-42e8-a238-2b363ad846cb-kube-api-access-7b9hf\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:41:24.624777 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:24.624779 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a975b396-b01f-42e8-a238-2b363ad846cb-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:41:25.253679 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:25.253651 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph" Oct 30 19:41:25.253861 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:25.253666 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph" event={"ID":"a975b396-b01f-42e8-a238-2b363ad846cb","Type":"ContainerDied","Data":"0339edd0654cc5c8ef55397df1a57c8adf05e222cf4af4b6155b3631ca7e4507"} Oct 30 19:41:25.253861 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:25.253736 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0339edd0654cc5c8ef55397df1a57c8adf05e222cf4af4b6155b3631ca7e4507" Oct 30 19:41:26.400114 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:41:26.400073 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:41:34.237482 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:34.237434 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb"] Oct 30 19:41:34.286489 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:34.286455 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-23302c-rmjkb"] Oct 30 19:41:34.885742 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:34.885691 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb"] Oct 30 19:41:35.438849 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:35.438798 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p"] Oct 30 19:41:35.442164 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:35.442105 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-dc5cdd-btw9p"] Oct 30 19:41:36.102646 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:36.102608 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7649797-82f3-4199-9d40-d2e802a3eac9" path="/var/lib/kubelet/pods/a7649797-82f3-4199-9d40-d2e802a3eac9/volumes" Oct 30 19:41:36.103041 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:36.103027 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f29b3c44-0710-41cd-983f-553d775f2593" path="/var/lib/kubelet/pods/f29b3c44-0710-41cd-983f-553d775f2593/volumes" Oct 30 19:41:39.886195 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:39.886116 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb" podUID="bc9144a3-39f6-4066-b677-0261f5f9fe3e" containerName="payload-pre-processing" containerID="cri-o://d26ea88bdb144b2cea10d8d390815b9159114e6afe5586d69e2c3e33649e5727" gracePeriod=25 Oct 30 19:41:43.780333 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:43.780286 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x"] Oct 30 19:41:43.780911 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:43.780541 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" podUID="0484c6c5-9f9e-408c-9974-a4f5f85b6bea" containerName="istio-proxy" containerID="cri-o://41cfa381c5667829bb696c92ceee114df2d57b84febf90206e4b618b96d822e2" gracePeriod=30 Oct 30 19:41:43.820019 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:43.819972 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4"] Oct 30 19:41:43.820551 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:43.820523 4531 cpu_manager.go:414] "containerMap: removing container" podUID="eb431a4c-7eba-4bf5-ae56-56cba97e5342" containerName="payload-processing" Oct 30 19:41:43.820551 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:43.820546 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb431a4c-7eba-4bf5-ae56-56cba97e5342" containerName="payload-processing" Oct 30 19:41:43.820725 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:43.820578 4531 cpu_manager.go:414] "containerMap: removing container" podUID="cc0b354d-95e7-477f-b479-2028dcc5a950" containerName="payload-pre-processing" Oct 30 19:41:43.820725 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:43.820587 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0b354d-95e7-477f-b479-2028dcc5a950" containerName="payload-pre-processing" Oct 30 19:41:43.820725 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:43.820599 4531 cpu_manager.go:414] "containerMap: removing container" podUID="a975b396-b01f-42e8-a238-2b363ad846cb" containerName="revoke-keys" Oct 30 19:41:43.820725 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:43.820604 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a975b396-b01f-42e8-a238-2b363ad846cb" containerName="revoke-keys" Oct 30 19:41:43.820725 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:43.820681 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a975b396-b01f-42e8-a238-2b363ad846cb" containerName="revoke-keys" Oct 30 19:41:43.820725 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:43.820694 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb431a4c-7eba-4bf5-ae56-56cba97e5342" containerName="payload-processing" Oct 30 19:41:43.820725 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:43.820704 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc0b354d-95e7-477f-b479-2028dcc5a950" containerName="payload-pre-processing" Oct 30 19:41:43.824043 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:43.824019 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4" Oct 30 19:41:43.876689 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:43.876656 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:41:43.876921 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:43.876725 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:41:43.876921 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:43.876849 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:41:43.876921 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:43.876897 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:41:43.883923 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:43.883874 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4"] Oct 30 19:41:43.949634 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:43.949584 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa39ec7-9f92-411e-aa6f-209aa15b43a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4\" (UID: \"daa39ec7-9f92-411e-aa6f-209aa15b43a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4" Oct 30 19:41:43.949860 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:43.949644 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bclq\" (UniqueName: \"kubernetes.io/projected/daa39ec7-9f92-411e-aa6f-209aa15b43a2-kube-api-access-6bclq\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4\" (UID: \"daa39ec7-9f92-411e-aa6f-209aa15b43a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4" Oct 30 19:41:43.949860 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:43.949828 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa39ec7-9f92-411e-aa6f-209aa15b43a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4\" (UID: \"daa39ec7-9f92-411e-aa6f-209aa15b43a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4" Oct 30 19:41:44.051211 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:44.051100 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa39ec7-9f92-411e-aa6f-209aa15b43a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4\" (UID: \"daa39ec7-9f92-411e-aa6f-209aa15b43a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4" Oct 30 19:41:44.051415 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:44.051234 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bclq\" (UniqueName: \"kubernetes.io/projected/daa39ec7-9f92-411e-aa6f-209aa15b43a2-kube-api-access-6bclq\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4\" (UID: \"daa39ec7-9f92-411e-aa6f-209aa15b43a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4" Oct 30 19:41:44.051415 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:44.051341 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa39ec7-9f92-411e-aa6f-209aa15b43a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4\" (UID: \"daa39ec7-9f92-411e-aa6f-209aa15b43a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4" Oct 30 19:41:44.051818 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:44.051782 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa39ec7-9f92-411e-aa6f-209aa15b43a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4\" (UID: \"daa39ec7-9f92-411e-aa6f-209aa15b43a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4" Oct 30 19:41:44.054035 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:44.054010 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa39ec7-9f92-411e-aa6f-209aa15b43a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4\" (UID: \"daa39ec7-9f92-411e-aa6f-209aa15b43a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4" Oct 30 19:41:44.113114 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:44.113067 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bclq\" (UniqueName: \"kubernetes.io/projected/daa39ec7-9f92-411e-aa6f-209aa15b43a2-kube-api-access-6bclq\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4\" (UID: \"daa39ec7-9f92-411e-aa6f-209aa15b43a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4" Oct 30 19:41:44.138806 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:44.138763 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4" Oct 30 19:41:44.521979 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:44.521952 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4"] Oct 30 19:41:44.524261 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:41:44.524230 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa39ec7_9f92_411e_aa6f_209aa15b43a2.slice/crio-0db5f40521c66e6405501ebf6bfd5e06da944ed4d3893429c8a73df773ee5703 WatchSource:0}: Error finding container 0db5f40521c66e6405501ebf6bfd5e06da944ed4d3893429c8a73df773ee5703: Status 404 returned error can't find the container with id 0db5f40521c66e6405501ebf6bfd5e06da944ed4d3893429c8a73df773ee5703 Oct 30 19:41:44.884170 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:44.884087 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4"] Oct 30 19:41:44.887968 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:44.887939 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4" Oct 30 19:41:44.949683 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:44.949625 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4"] Oct 30 19:41:45.062895 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:45.062863 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/498eff0d-6631-4255-a0f4-8491a9174258-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4\" (UID: \"498eff0d-6631-4255-a0f4-8491a9174258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4" Oct 30 19:41:45.063082 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:45.062908 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l66wh\" (UniqueName: \"kubernetes.io/projected/498eff0d-6631-4255-a0f4-8491a9174258-kube-api-access-l66wh\") pod \"maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4\" (UID: \"498eff0d-6631-4255-a0f4-8491a9174258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4" Oct 30 19:41:45.063082 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:45.063001 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/498eff0d-6631-4255-a0f4-8491a9174258-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4\" (UID: \"498eff0d-6631-4255-a0f4-8491a9174258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4" Oct 30 19:41:45.164364 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:45.164273 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/498eff0d-6631-4255-a0f4-8491a9174258-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4\" (UID: \"498eff0d-6631-4255-a0f4-8491a9174258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4" Oct 30 19:41:45.164364 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:45.164326 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l66wh\" (UniqueName: \"kubernetes.io/projected/498eff0d-6631-4255-a0f4-8491a9174258-kube-api-access-l66wh\") pod \"maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4\" (UID: \"498eff0d-6631-4255-a0f4-8491a9174258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4" Oct 30 19:41:45.164364 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:45.164363 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/498eff0d-6631-4255-a0f4-8491a9174258-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4\" (UID: \"498eff0d-6631-4255-a0f4-8491a9174258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4" Oct 30 19:41:45.165103 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:45.165081 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/498eff0d-6631-4255-a0f4-8491a9174258-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4\" (UID: \"498eff0d-6631-4255-a0f4-8491a9174258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4" Oct 30 19:41:45.166839 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:45.166818 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/498eff0d-6631-4255-a0f4-8491a9174258-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4\" (UID: \"498eff0d-6631-4255-a0f4-8491a9174258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4" Oct 30 19:41:45.226569 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:45.226528 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l66wh\" (UniqueName: \"kubernetes.io/projected/498eff0d-6631-4255-a0f4-8491a9174258-kube-api-access-l66wh\") pod \"maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4\" (UID: \"498eff0d-6631-4255-a0f4-8491a9174258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4" Oct 30 19:41:45.398977 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:45.398934 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4" event={"ID":"daa39ec7-9f92-411e-aa6f-209aa15b43a2","Type":"ContainerStarted","Data":"96e7a32d9eeaec79600dc42d05a0143f9a6257e41065cf1cfacba6acc8476b26"} Oct 30 19:41:45.399189 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:45.399002 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4" event={"ID":"daa39ec7-9f92-411e-aa6f-209aa15b43a2","Type":"ContainerStarted","Data":"0db5f40521c66e6405501ebf6bfd5e06da944ed4d3893429c8a73df773ee5703"} Oct 30 19:41:45.501842 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:45.501798 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4" Oct 30 19:41:45.516861 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:45.516795 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4" podStartSLOduration=2.51677307 podStartE2EDuration="2.51677307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:45.515581567 +0000 UTC m=+4065.542686527" watchObservedRunningTime="2026-10-30 19:41:45.51677307 +0000 UTC m=+4065.543878028" Oct 30 19:41:45.883929 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:45.883902 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4"] Oct 30 19:41:45.886033 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:41:45.886002 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod498eff0d_6631_4255_a0f4_8491a9174258.slice/crio-9b67281ccfac47fa6e80aaaf7d776037bb193328b31285c993b8713cd7e31136 WatchSource:0}: Error finding container 9b67281ccfac47fa6e80aaaf7d776037bb193328b31285c993b8713cd7e31136: Status 404 returned error can't find the container with id 9b67281ccfac47fa6e80aaaf7d776037bb193328b31285c993b8713cd7e31136 Oct 30 19:41:46.406835 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:46.406794 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4" event={"ID":"498eff0d-6631-4255-a0f4-8491a9174258","Type":"ContainerStarted","Data":"c50a19d3684c561e9732897feec5a1afbfd959cd03ee9f07e1c4aab6a7366839"} Oct 30 19:41:46.407060 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:46.406863 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4" event={"ID":"498eff0d-6631-4255-a0f4-8491a9174258","Type":"ContainerStarted","Data":"9b67281ccfac47fa6e80aaaf7d776037bb193328b31285c993b8713cd7e31136"} Oct 30 19:41:46.408716 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:46.408686 4531 generic.go:356] "Generic (PLEG): container finished" podID="daa39ec7-9f92-411e-aa6f-209aa15b43a2" containerID="96e7a32d9eeaec79600dc42d05a0143f9a6257e41065cf1cfacba6acc8476b26" exitCode=0 Oct 30 19:41:46.408848 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:46.408764 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4" event={"ID":"daa39ec7-9f92-411e-aa6f-209aa15b43a2","Type":"ContainerDied","Data":"96e7a32d9eeaec79600dc42d05a0143f9a6257e41065cf1cfacba6acc8476b26"} Oct 30 19:41:46.526619 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:46.526567 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4" podStartSLOduration=2.52655027 podStartE2EDuration="2.52655027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:46.523320865 +0000 UTC m=+4066.550425824" watchObservedRunningTime="2026-10-30 19:41:46.52655027 +0000 UTC m=+4066.553655228" Oct 30 19:41:47.418070 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:47.418034 4531 generic.go:356] "Generic (PLEG): container finished" podID="498eff0d-6631-4255-a0f4-8491a9174258" containerID="c50a19d3684c561e9732897feec5a1afbfd959cd03ee9f07e1c4aab6a7366839" exitCode=0 Oct 30 19:41:47.418618 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:47.418155 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4" event={"ID":"498eff0d-6631-4255-a0f4-8491a9174258","Type":"ContainerDied","Data":"c50a19d3684c561e9732897feec5a1afbfd959cd03ee9f07e1c4aab6a7366839"} Oct 30 19:41:47.590788 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:47.590750 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4" Oct 30 19:41:47.593149 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:47.593095 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daa39ec7-9f92-411e-aa6f-209aa15b43a2-kube-api-access-6bclq\" (UniqueName: \"kubernetes.io/projected/daa39ec7-9f92-411e-aa6f-209aa15b43a2-kube-api-access-6bclq\") pod \"daa39ec7-9f92-411e-aa6f-209aa15b43a2\" (UID: \"daa39ec7-9f92-411e-aa6f-209aa15b43a2\") " Oct 30 19:41:47.593341 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:47.593232 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daa39ec7-9f92-411e-aa6f-209aa15b43a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa39ec7-9f92-411e-aa6f-209aa15b43a2-maas-api-openshift-ca\") pod \"daa39ec7-9f92-411e-aa6f-209aa15b43a2\" (UID: \"daa39ec7-9f92-411e-aa6f-209aa15b43a2\") " Oct 30 19:41:47.593341 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:47.593281 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daa39ec7-9f92-411e-aa6f-209aa15b43a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa39ec7-9f92-411e-aa6f-209aa15b43a2-maas-api-xks-ca\") pod \"daa39ec7-9f92-411e-aa6f-209aa15b43a2\" (UID: \"daa39ec7-9f92-411e-aa6f-209aa15b43a2\") " Oct 30 19:41:47.594052 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:47.594022 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daa39ec7-9f92-411e-aa6f-209aa15b43a2-maas-api-openshift-ca" pod "daa39ec7-9f92-411e-aa6f-209aa15b43a2" (UID: "daa39ec7-9f92-411e-aa6f-209aa15b43a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:47.596109 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:47.596076 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa39ec7-9f92-411e-aa6f-209aa15b43a2-kube-api-access-6bclq" pod "daa39ec7-9f92-411e-aa6f-209aa15b43a2" (UID: "daa39ec7-9f92-411e-aa6f-209aa15b43a2"). InnerVolumeSpecName "kube-api-access-6bclq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:47.596364 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:47.596340 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daa39ec7-9f92-411e-aa6f-209aa15b43a2-maas-api-xks-ca" pod "daa39ec7-9f92-411e-aa6f-209aa15b43a2" (UID: "daa39ec7-9f92-411e-aa6f-209aa15b43a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:41:47.694816 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:47.694486 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa39ec7-9f92-411e-aa6f-209aa15b43a2-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:41:47.694816 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:47.694527 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa39ec7-9f92-411e-aa6f-209aa15b43a2-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:41:47.694816 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:47.694545 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bclq\" (UniqueName: \"kubernetes.io/projected/daa39ec7-9f92-411e-aa6f-209aa15b43a2-kube-api-access-6bclq\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:41:48.425528 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:48.425433 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4" Oct 30 19:41:48.425528 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:48.425433 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4" event={"ID":"daa39ec7-9f92-411e-aa6f-209aa15b43a2","Type":"ContainerDied","Data":"0db5f40521c66e6405501ebf6bfd5e06da944ed4d3893429c8a73df773ee5703"} Oct 30 19:41:48.425528 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:48.425513 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0db5f40521c66e6405501ebf6bfd5e06da944ed4d3893429c8a73df773ee5703" Oct 30 19:41:48.567115 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:48.567074 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4" Oct 30 19:41:48.603601 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:48.603560 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/498eff0d-6631-4255-a0f4-8491a9174258-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/498eff0d-6631-4255-a0f4-8491a9174258-maas-api-xks-ca\") pod \"498eff0d-6631-4255-a0f4-8491a9174258\" (UID: \"498eff0d-6631-4255-a0f4-8491a9174258\") " Oct 30 19:41:48.603806 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:48.603685 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/498eff0d-6631-4255-a0f4-8491a9174258-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/498eff0d-6631-4255-a0f4-8491a9174258-maas-api-openshift-ca\") pod \"498eff0d-6631-4255-a0f4-8491a9174258\" (UID: \"498eff0d-6631-4255-a0f4-8491a9174258\") " Oct 30 19:41:48.603806 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:48.603776 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/498eff0d-6631-4255-a0f4-8491a9174258-kube-api-access-l66wh\" (UniqueName: \"kubernetes.io/projected/498eff0d-6631-4255-a0f4-8491a9174258-kube-api-access-l66wh\") pod \"498eff0d-6631-4255-a0f4-8491a9174258\" (UID: \"498eff0d-6631-4255-a0f4-8491a9174258\") " Oct 30 19:41:48.604078 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:48.604048 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/498eff0d-6631-4255-a0f4-8491a9174258-maas-api-openshift-ca" pod "498eff0d-6631-4255-a0f4-8491a9174258" (UID: "498eff0d-6631-4255-a0f4-8491a9174258"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:48.604252 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:48.604214 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/498eff0d-6631-4255-a0f4-8491a9174258-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:41:48.606054 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:48.606023 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/498eff0d-6631-4255-a0f4-8491a9174258-maas-api-xks-ca" pod "498eff0d-6631-4255-a0f4-8491a9174258" (UID: "498eff0d-6631-4255-a0f4-8491a9174258"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:41:48.606245 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:48.606173 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/498eff0d-6631-4255-a0f4-8491a9174258-kube-api-access-l66wh" pod "498eff0d-6631-4255-a0f4-8491a9174258" (UID: "498eff0d-6631-4255-a0f4-8491a9174258"). InnerVolumeSpecName "kube-api-access-l66wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:48.704905 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:48.704817 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/498eff0d-6631-4255-a0f4-8491a9174258-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:41:48.704905 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:48.704851 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l66wh\" (UniqueName: \"kubernetes.io/projected/498eff0d-6631-4255-a0f4-8491a9174258-kube-api-access-l66wh\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:41:49.040887 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.040853 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:41:49.112315 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.112278 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-envoy\") pod \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " Oct 30 19:41:49.112520 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.112331 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-data\") pod \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " Oct 30 19:41:49.112520 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.112370 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-workload-socket\") pod \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " Oct 30 19:41:49.112520 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.112391 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-podinfo\") pod \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " Oct 30 19:41:49.112520 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.112485 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-kube-api-access-88sz7\" (UniqueName: \"kubernetes.io/projected/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-kube-api-access-88sz7\") pod \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " Oct 30 19:41:49.112520 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.112512 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istiod-ca-cert\") pod \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " Oct 30 19:41:49.112821 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.112535 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-workload-certs\") pod \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " Oct 30 19:41:49.112821 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.112613 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-credential-socket\") pod \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " Oct 30 19:41:49.112821 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.112653 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-token\" (UniqueName: \"kubernetes.io/projected/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-token\") pod \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\" (UID: \"0484c6c5-9f9e-408c-9974-a4f5f85b6bea\") " Oct 30 19:41:49.112821 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.112678 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-data" pod "0484c6c5-9f9e-408c-9974-a4f5f85b6bea" (UID: "0484c6c5-9f9e-408c-9974-a4f5f85b6bea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:49.112821 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.112710 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-workload-socket" pod "0484c6c5-9f9e-408c-9974-a4f5f85b6bea" (UID: "0484c6c5-9f9e-408c-9974-a4f5f85b6bea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:49.113121 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.112932 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-credential-socket" pod "0484c6c5-9f9e-408c-9974-a4f5f85b6bea" (UID: "0484c6c5-9f9e-408c-9974-a4f5f85b6bea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:49.113121 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.112941 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istiod-ca-cert" pod "0484c6c5-9f9e-408c-9974-a4f5f85b6bea" (UID: "0484c6c5-9f9e-408c-9974-a4f5f85b6bea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:49.113457 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.113178 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-workload-certs" pod "0484c6c5-9f9e-408c-9974-a4f5f85b6bea" (UID: "0484c6c5-9f9e-408c-9974-a4f5f85b6bea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:49.113457 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.113249 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:41:49.113457 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.113271 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:41:49.113457 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.113288 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:41:49.113457 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.113303 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:41:49.115052 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.115022 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-kube-api-access-88sz7" pod "0484c6c5-9f9e-408c-9974-a4f5f85b6bea" (UID: "0484c6c5-9f9e-408c-9974-a4f5f85b6bea"). InnerVolumeSpecName "kube-api-access-88sz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:49.115432 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.115411 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-podinfo" pod "0484c6c5-9f9e-408c-9974-a4f5f85b6bea" (UID: "0484c6c5-9f9e-408c-9974-a4f5f85b6bea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:41:49.115518 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.115500 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-token" pod "0484c6c5-9f9e-408c-9974-a4f5f85b6bea" (UID: "0484c6c5-9f9e-408c-9974-a4f5f85b6bea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:49.115556 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.115525 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-envoy" pod "0484c6c5-9f9e-408c-9974-a4f5f85b6bea" (UID: "0484c6c5-9f9e-408c-9974-a4f5f85b6bea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:49.213995 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.213943 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:41:49.213995 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.213984 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:41:49.213995 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.213999 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:41:49.214312 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.214011 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:41:49.214312 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.214027 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88sz7\" (UniqueName: \"kubernetes.io/projected/0484c6c5-9f9e-408c-9974-a4f5f85b6bea-kube-api-access-88sz7\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:41:49.264673 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.264635 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb" podUID="bc9144a3-39f6-4066-b677-0261f5f9fe3e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:41:49.287497 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.287407 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj"] Oct 30 19:41:49.433002 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.432974 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4" Oct 30 19:41:49.433564 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.433014 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4" event={"ID":"498eff0d-6631-4255-a0f4-8491a9174258","Type":"ContainerDied","Data":"9b67281ccfac47fa6e80aaaf7d776037bb193328b31285c993b8713cd7e31136"} Oct 30 19:41:49.433564 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.433072 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b67281ccfac47fa6e80aaaf7d776037bb193328b31285c993b8713cd7e31136" Oct 30 19:41:49.435119 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.435092 4531 generic.go:356] "Generic (PLEG): container finished" podID="0484c6c5-9f9e-408c-9974-a4f5f85b6bea" containerID="41cfa381c5667829bb696c92ceee114df2d57b84febf90206e4b618b96d822e2" exitCode=0 Oct 30 19:41:49.435299 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.435163 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" Oct 30 19:41:49.435299 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.435168 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" event={"ID":"0484c6c5-9f9e-408c-9974-a4f5f85b6bea","Type":"ContainerDied","Data":"41cfa381c5667829bb696c92ceee114df2d57b84febf90206e4b618b96d822e2"} Oct 30 19:41:49.435299 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.435224 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x" event={"ID":"0484c6c5-9f9e-408c-9974-a4f5f85b6bea","Type":"ContainerDied","Data":"c7c7087b07f87ccf59627c2b4ce7dc213d7e81cbb2790b0e6475fe887da00ede"} Oct 30 19:41:49.435299 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.435247 4531 scope.go:122] "RemoveContainer" containerID="41cfa381c5667829bb696c92ceee114df2d57b84febf90206e4b618b96d822e2" Oct 30 19:41:49.446062 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.446037 4531 scope.go:122] "RemoveContainer" containerID="a41389ff7303cf57aa56f4ba914e97864d722a33b72ab90c2a940309552f90bb" Oct 30 19:41:49.456042 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.456020 4531 scope.go:122] "RemoveContainer" containerID="41cfa381c5667829bb696c92ceee114df2d57b84febf90206e4b618b96d822e2" Oct 30 19:41:49.456457 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:41:49.456431 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41cfa381c5667829bb696c92ceee114df2d57b84febf90206e4b618b96d822e2\": container with ID starting with 41cfa381c5667829bb696c92ceee114df2d57b84febf90206e4b618b96d822e2 not found: ID does not exist" containerID="41cfa381c5667829bb696c92ceee114df2d57b84febf90206e4b618b96d822e2" Oct 30 19:41:49.456530 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.456468 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41cfa381c5667829bb696c92ceee114df2d57b84febf90206e4b618b96d822e2"} err="failed to get container status \"41cfa381c5667829bb696c92ceee114df2d57b84febf90206e4b618b96d822e2\": rpc error: code = NotFound desc = could not find container \"41cfa381c5667829bb696c92ceee114df2d57b84febf90206e4b618b96d822e2\": container with ID starting with 41cfa381c5667829bb696c92ceee114df2d57b84febf90206e4b618b96d822e2 not found: ID does not exist" Oct 30 19:41:49.456530 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.456490 4531 scope.go:122] "RemoveContainer" containerID="a41389ff7303cf57aa56f4ba914e97864d722a33b72ab90c2a940309552f90bb" Oct 30 19:41:49.456784 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:41:49.456768 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a41389ff7303cf57aa56f4ba914e97864d722a33b72ab90c2a940309552f90bb\": container with ID starting with a41389ff7303cf57aa56f4ba914e97864d722a33b72ab90c2a940309552f90bb not found: ID does not exist" containerID="a41389ff7303cf57aa56f4ba914e97864d722a33b72ab90c2a940309552f90bb" Oct 30 19:41:49.456853 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.456790 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a41389ff7303cf57aa56f4ba914e97864d722a33b72ab90c2a940309552f90bb"} err="failed to get container status \"a41389ff7303cf57aa56f4ba914e97864d722a33b72ab90c2a940309552f90bb\": rpc error: code = NotFound desc = could not find container \"a41389ff7303cf57aa56f4ba914e97864d722a33b72ab90c2a940309552f90bb\": container with ID starting with a41389ff7303cf57aa56f4ba914e97864d722a33b72ab90c2a940309552f90bb not found: ID does not exist" Oct 30 19:41:49.600165 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.600106 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x"] Oct 30 19:41:49.605847 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:49.605810 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-dc5cdd-openshift-default-6db445f47f-7hs5x"] Oct 30 19:41:50.099111 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:50.099076 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0484c6c5-9f9e-408c-9974-a4f5f85b6bea" path="/var/lib/kubelet/pods/0484c6c5-9f9e-408c-9974-a4f5f85b6bea/volumes" Oct 30 19:41:52.514424 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:52.514387 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph"] Oct 30 19:41:52.563873 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:52.563828 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-19f6ea-gwcph"] Oct 30 19:41:54.098651 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:54.098611 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a975b396-b01f-42e8-a238-2b363ad846cb" path="/var/lib/kubelet/pods/a975b396-b01f-42e8-a238-2b363ad846cb/volumes" Oct 30 19:41:55.367172 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:55.367117 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v"] Oct 30 19:41:56.398828 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:41:56.398793 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:41:59.263400 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:41:59.263358 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb" podUID="bc9144a3-39f6-4066-b677-0261f5f9fe3e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:42:00.161425 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:00.161347 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb" Oct 30 19:42:00.236119 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:00.236079 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc9144a3-39f6-4066-b677-0261f5f9fe3e-kube-api-access-2znlx\" (UniqueName: \"kubernetes.io/projected/bc9144a3-39f6-4066-b677-0261f5f9fe3e-kube-api-access-2znlx\") pod \"bc9144a3-39f6-4066-b677-0261f5f9fe3e\" (UID: \"bc9144a3-39f6-4066-b677-0261f5f9fe3e\") " Oct 30 19:42:00.236347 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:00.236257 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc9144a3-39f6-4066-b677-0261f5f9fe3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9144a3-39f6-4066-b677-0261f5f9fe3e-plugins-config-volume\") pod \"bc9144a3-39f6-4066-b677-0261f5f9fe3e\" (UID: \"bc9144a3-39f6-4066-b677-0261f5f9fe3e\") " Oct 30 19:42:00.236653 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:00.236627 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc9144a3-39f6-4066-b677-0261f5f9fe3e-plugins-config-volume" pod "bc9144a3-39f6-4066-b677-0261f5f9fe3e" (UID: "bc9144a3-39f6-4066-b677-0261f5f9fe3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:00.238512 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:00.238485 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc9144a3-39f6-4066-b677-0261f5f9fe3e-kube-api-access-2znlx" pod "bc9144a3-39f6-4066-b677-0261f5f9fe3e" (UID: "bc9144a3-39f6-4066-b677-0261f5f9fe3e"). InnerVolumeSpecName "kube-api-access-2znlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:00.337795 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:00.337755 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2znlx\" (UniqueName: \"kubernetes.io/projected/bc9144a3-39f6-4066-b677-0261f5f9fe3e-kube-api-access-2znlx\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:00.338374 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:00.338343 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9144a3-39f6-4066-b677-0261f5f9fe3e-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:00.368120 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:00.368079 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v" podUID="06335665-17e4-4b6a-b7a6-30ef62625cfe" containerName="payload-processing" containerID="cri-o://4163e63fc110f331c0d6d095431820d6ae72f53e474a82e0df07ce10498e43a5" gracePeriod=25 Oct 30 19:42:00.506340 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:00.506292 4531 generic.go:356] "Generic (PLEG): container finished" podID="bc9144a3-39f6-4066-b677-0261f5f9fe3e" containerID="d26ea88bdb144b2cea10d8d390815b9159114e6afe5586d69e2c3e33649e5727" exitCode=0 Oct 30 19:42:00.506535 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:00.506408 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb" Oct 30 19:42:00.506535 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:00.506428 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb" event={"ID":"bc9144a3-39f6-4066-b677-0261f5f9fe3e","Type":"ContainerDied","Data":"d26ea88bdb144b2cea10d8d390815b9159114e6afe5586d69e2c3e33649e5727"} Oct 30 19:42:00.506535 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:00.506490 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb" event={"ID":"bc9144a3-39f6-4066-b677-0261f5f9fe3e","Type":"ContainerDied","Data":"954a7e255d4c7293bf6f45c0edfa0c02a37a5e60257c0faf8ec18d84986cafe8"} Oct 30 19:42:00.506535 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:00.506515 4531 scope.go:122] "RemoveContainer" containerID="d26ea88bdb144b2cea10d8d390815b9159114e6afe5586d69e2c3e33649e5727" Oct 30 19:42:00.517669 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:00.517650 4531 scope.go:122] "RemoveContainer" containerID="d26ea88bdb144b2cea10d8d390815b9159114e6afe5586d69e2c3e33649e5727" Oct 30 19:42:00.518014 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:42:00.517993 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d26ea88bdb144b2cea10d8d390815b9159114e6afe5586d69e2c3e33649e5727\": container with ID starting with d26ea88bdb144b2cea10d8d390815b9159114e6afe5586d69e2c3e33649e5727 not found: ID does not exist" containerID="d26ea88bdb144b2cea10d8d390815b9159114e6afe5586d69e2c3e33649e5727" Oct 30 19:42:00.518073 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:00.518022 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d26ea88bdb144b2cea10d8d390815b9159114e6afe5586d69e2c3e33649e5727"} err="failed to get container status \"d26ea88bdb144b2cea10d8d390815b9159114e6afe5586d69e2c3e33649e5727\": rpc error: code = NotFound desc = could not find container \"d26ea88bdb144b2cea10d8d390815b9159114e6afe5586d69e2c3e33649e5727\": container with ID starting with d26ea88bdb144b2cea10d8d390815b9159114e6afe5586d69e2c3e33649e5727 not found: ID does not exist" Oct 30 19:42:00.672620 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:00.672585 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb"] Oct 30 19:42:00.675790 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:00.675756 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-19f6ea-5dd9876f67-54wzb"] Oct 30 19:42:01.248647 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.248607 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc"] Oct 30 19:42:01.249062 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.249037 4531 cpu_manager.go:414] "containerMap: removing container" podUID="bc9144a3-39f6-4066-b677-0261f5f9fe3e" containerName="payload-pre-processing" Oct 30 19:42:01.249062 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.249055 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc9144a3-39f6-4066-b677-0261f5f9fe3e" containerName="payload-pre-processing" Oct 30 19:42:01.249062 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.249067 4531 cpu_manager.go:414] "containerMap: removing container" podUID="0484c6c5-9f9e-408c-9974-a4f5f85b6bea" containerName="istio-proxy" Oct 30 19:42:01.249308 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.249073 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0484c6c5-9f9e-408c-9974-a4f5f85b6bea" containerName="istio-proxy" Oct 30 19:42:01.249308 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.249088 4531 cpu_manager.go:414] "containerMap: removing container" podUID="daa39ec7-9f92-411e-aa6f-209aa15b43a2" containerName="revoke-keys" Oct 30 19:42:01.249308 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.249095 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daa39ec7-9f92-411e-aa6f-209aa15b43a2" containerName="revoke-keys" Oct 30 19:42:01.249308 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.249119 4531 cpu_manager.go:414] "containerMap: removing container" podUID="0484c6c5-9f9e-408c-9974-a4f5f85b6bea" containerName="istio-proxy" Oct 30 19:42:01.249308 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.249141 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0484c6c5-9f9e-408c-9974-a4f5f85b6bea" containerName="istio-proxy" Oct 30 19:42:01.249308 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.249170 4531 cpu_manager.go:414] "containerMap: removing container" podUID="498eff0d-6631-4255-a0f4-8491a9174258" containerName="revoke-keys" Oct 30 19:42:01.249308 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.249177 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="498eff0d-6631-4255-a0f4-8491a9174258" containerName="revoke-keys" Oct 30 19:42:01.249308 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.249260 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc9144a3-39f6-4066-b677-0261f5f9fe3e" containerName="payload-pre-processing" Oct 30 19:42:01.249308 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.249271 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0484c6c5-9f9e-408c-9974-a4f5f85b6bea" containerName="istio-proxy" Oct 30 19:42:01.249308 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.249281 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="daa39ec7-9f92-411e-aa6f-209aa15b43a2" containerName="revoke-keys" Oct 30 19:42:01.249308 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.249289 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0484c6c5-9f9e-408c-9974-a4f5f85b6bea" containerName="istio-proxy" Oct 30 19:42:01.249308 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.249299 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="498eff0d-6631-4255-a0f4-8491a9174258" containerName="revoke-keys" Oct 30 19:42:01.254006 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.253980 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc" Oct 30 19:42:01.311553 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.311516 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc"] Oct 30 19:42:01.350379 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.350339 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c63b358-0b01-466a-8bc6-88771b1a86bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc\" (UID: \"5c63b358-0b01-466a-8bc6-88771b1a86bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc" Oct 30 19:42:01.350902 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.350403 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c63b358-0b01-466a-8bc6-88771b1a86bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc\" (UID: \"5c63b358-0b01-466a-8bc6-88771b1a86bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc" Oct 30 19:42:01.350902 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.350624 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5swtz\" (UniqueName: \"kubernetes.io/projected/5c63b358-0b01-466a-8bc6-88771b1a86bd-kube-api-access-5swtz\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc\" (UID: \"5c63b358-0b01-466a-8bc6-88771b1a86bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc" Oct 30 19:42:01.358436 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.358385 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:42:01.358436 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.358393 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:42:01.358682 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.358392 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:42:01.358682 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.358454 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:42:01.451420 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.451382 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5swtz\" (UniqueName: \"kubernetes.io/projected/5c63b358-0b01-466a-8bc6-88771b1a86bd-kube-api-access-5swtz\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc\" (UID: \"5c63b358-0b01-466a-8bc6-88771b1a86bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc" Oct 30 19:42:01.451635 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.451448 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c63b358-0b01-466a-8bc6-88771b1a86bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc\" (UID: \"5c63b358-0b01-466a-8bc6-88771b1a86bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc" Oct 30 19:42:01.451635 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.451481 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c63b358-0b01-466a-8bc6-88771b1a86bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc\" (UID: \"5c63b358-0b01-466a-8bc6-88771b1a86bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc" Oct 30 19:42:01.452229 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.452204 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c63b358-0b01-466a-8bc6-88771b1a86bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc\" (UID: \"5c63b358-0b01-466a-8bc6-88771b1a86bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc" Oct 30 19:42:01.454099 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.454073 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c63b358-0b01-466a-8bc6-88771b1a86bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc\" (UID: \"5c63b358-0b01-466a-8bc6-88771b1a86bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc" Oct 30 19:42:01.513482 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.513398 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5swtz\" (UniqueName: \"kubernetes.io/projected/5c63b358-0b01-466a-8bc6-88771b1a86bd-kube-api-access-5swtz\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc\" (UID: \"5c63b358-0b01-466a-8bc6-88771b1a86bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc" Oct 30 19:42:01.568247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.568207 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc" Oct 30 19:42:01.947262 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:01.947238 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc"] Oct 30 19:42:01.949698 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:42:01.949662 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c63b358_0b01_466a_8bc6_88771b1a86bd.slice/crio-7fe96caedfb0984accdb4d6d3ca3a49f972e483ab7a2b4ce9e7cb93820656646 WatchSource:0}: Error finding container 7fe96caedfb0984accdb4d6d3ca3a49f972e483ab7a2b4ce9e7cb93820656646: Status 404 returned error can't find the container with id 7fe96caedfb0984accdb4d6d3ca3a49f972e483ab7a2b4ce9e7cb93820656646 Oct 30 19:42:02.101861 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:02.101824 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc9144a3-39f6-4066-b677-0261f5f9fe3e" path="/var/lib/kubelet/pods/bc9144a3-39f6-4066-b677-0261f5f9fe3e/volumes" Oct 30 19:42:02.272560 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:02.272508 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v" podUID="06335665-17e4-4b6a-b7a6-30ef62625cfe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:42:02.521205 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:02.521161 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc" event={"ID":"5c63b358-0b01-466a-8bc6-88771b1a86bd","Type":"ContainerStarted","Data":"d55863bf73355f3955caaad8db4d59eb2e69c1ace70ee39c07b37b8526b5f100"} Oct 30 19:42:02.521620 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:02.521225 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc" event={"ID":"5c63b358-0b01-466a-8bc6-88771b1a86bd","Type":"ContainerStarted","Data":"7fe96caedfb0984accdb4d6d3ca3a49f972e483ab7a2b4ce9e7cb93820656646"} Oct 30 19:42:02.639252 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:02.639194 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc" podStartSLOduration=1.6391760359999998 podStartE2EDuration="1.639176036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:02.63662565 +0000 UTC m=+4082.663730610" watchObservedRunningTime="2026-10-30 19:42:02.639176036 +0000 UTC m=+4082.666280995" Oct 30 19:42:03.536522 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:03.536428 4531 generic.go:356] "Generic (PLEG): container finished" podID="5c63b358-0b01-466a-8bc6-88771b1a86bd" containerID="d55863bf73355f3955caaad8db4d59eb2e69c1ace70ee39c07b37b8526b5f100" exitCode=0 Oct 30 19:42:03.536896 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:03.536510 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc" event={"ID":"5c63b358-0b01-466a-8bc6-88771b1a86bd","Type":"ContainerDied","Data":"d55863bf73355f3955caaad8db4d59eb2e69c1ace70ee39c07b37b8526b5f100"} Oct 30 19:42:04.308084 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.308036 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" podUID="678454ff-4761-4bd4-b2a3-2b431bca7934" containerName="main" containerID="cri-o://e882a37b69b67c8744ed27e70afaf53f06c0d39d1bc7245cadd52111c312db60" gracePeriod=45 Oct 30 19:42:04.545608 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.545576 4531 generic.go:356] "Generic (PLEG): container finished" podID="678454ff-4761-4bd4-b2a3-2b431bca7934" containerID="e882a37b69b67c8744ed27e70afaf53f06c0d39d1bc7245cadd52111c312db60" exitCode=0 Oct 30 19:42:04.546045 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.545647 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" event={"ID":"678454ff-4761-4bd4-b2a3-2b431bca7934","Type":"ContainerDied","Data":"e882a37b69b67c8744ed27e70afaf53f06c0d39d1bc7245cadd52111c312db60"} Oct 30 19:42:04.571995 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.571917 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 19:42:04.684689 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.684659 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-dshm\" (UniqueName: \"kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-dshm\") pod \"678454ff-4761-4bd4-b2a3-2b431bca7934\" (UID: \"678454ff-4761-4bd4-b2a3-2b431bca7934\") " Oct 30 19:42:04.684689 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.684710 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/678454ff-4761-4bd4-b2a3-2b431bca7934-kube-api-access-j86jt\" (UniqueName: \"kubernetes.io/projected/678454ff-4761-4bd4-b2a3-2b431bca7934-kube-api-access-j86jt\") pod \"678454ff-4761-4bd4-b2a3-2b431bca7934\" (UID: \"678454ff-4761-4bd4-b2a3-2b431bca7934\") " Oct 30 19:42:04.684964 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.684755 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-home\" (UniqueName: \"kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-home\") pod \"678454ff-4761-4bd4-b2a3-2b431bca7934\" (UID: \"678454ff-4761-4bd4-b2a3-2b431bca7934\") " Oct 30 19:42:04.684964 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.684823 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/678454ff-4761-4bd4-b2a3-2b431bca7934-tls-certs\" (UniqueName: \"kubernetes.io/secret/678454ff-4761-4bd4-b2a3-2b431bca7934-tls-certs\") pod \"678454ff-4761-4bd4-b2a3-2b431bca7934\" (UID: \"678454ff-4761-4bd4-b2a3-2b431bca7934\") " Oct 30 19:42:04.684964 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.684880 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-model-cache\") pod \"678454ff-4761-4bd4-b2a3-2b431bca7934\" (UID: \"678454ff-4761-4bd4-b2a3-2b431bca7934\") " Oct 30 19:42:04.684964 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.684936 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-tmp-dir\") pod \"678454ff-4761-4bd4-b2a3-2b431bca7934\" (UID: \"678454ff-4761-4bd4-b2a3-2b431bca7934\") " Oct 30 19:42:04.685685 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.685312 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-model-cache" pod "678454ff-4761-4bd4-b2a3-2b431bca7934" (UID: "678454ff-4761-4bd4-b2a3-2b431bca7934"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:04.685685 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.685578 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-tmp-dir" pod "678454ff-4761-4bd4-b2a3-2b431bca7934" (UID: "678454ff-4761-4bd4-b2a3-2b431bca7934"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:04.685685 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.685660 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-home" pod "678454ff-4761-4bd4-b2a3-2b431bca7934" (UID: "678454ff-4761-4bd4-b2a3-2b431bca7934"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:04.687602 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.687570 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/678454ff-4761-4bd4-b2a3-2b431bca7934-kube-api-access-j86jt" pod "678454ff-4761-4bd4-b2a3-2b431bca7934" (UID: "678454ff-4761-4bd4-b2a3-2b431bca7934"). InnerVolumeSpecName "kube-api-access-j86jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:04.689814 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.689782 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/678454ff-4761-4bd4-b2a3-2b431bca7934-tls-certs" pod "678454ff-4761-4bd4-b2a3-2b431bca7934" (UID: "678454ff-4761-4bd4-b2a3-2b431bca7934"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:42:04.690086 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.690063 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-dshm" pod "678454ff-4761-4bd4-b2a3-2b431bca7934" (UID: "678454ff-4761-4bd4-b2a3-2b431bca7934"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:04.690568 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.690540 4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-dshm\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:04.690701 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.690573 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j86jt\" (UniqueName: \"kubernetes.io/projected/678454ff-4761-4bd4-b2a3-2b431bca7934-kube-api-access-j86jt\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:04.690701 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.690589 4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-home\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:04.690701 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.690603 4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/678454ff-4761-4bd4-b2a3-2b431bca7934-tls-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:04.690701 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.690618 4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-model-cache\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:04.690701 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.690659 4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/678454ff-4761-4bd4-b2a3-2b431bca7934-tmp-dir\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:04.722480 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.722447 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc" Oct 30 19:42:04.791887 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.791851 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c63b358-0b01-466a-8bc6-88771b1a86bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c63b358-0b01-466a-8bc6-88771b1a86bd-maas-api-openshift-ca\") pod \"5c63b358-0b01-466a-8bc6-88771b1a86bd\" (UID: \"5c63b358-0b01-466a-8bc6-88771b1a86bd\") " Oct 30 19:42:04.792086 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.791926 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c63b358-0b01-466a-8bc6-88771b1a86bd-kube-api-access-5swtz\" (UniqueName: \"kubernetes.io/projected/5c63b358-0b01-466a-8bc6-88771b1a86bd-kube-api-access-5swtz\") pod \"5c63b358-0b01-466a-8bc6-88771b1a86bd\" (UID: \"5c63b358-0b01-466a-8bc6-88771b1a86bd\") " Oct 30 19:42:04.792086 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.792019 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c63b358-0b01-466a-8bc6-88771b1a86bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c63b358-0b01-466a-8bc6-88771b1a86bd-maas-api-xks-ca\") pod \"5c63b358-0b01-466a-8bc6-88771b1a86bd\" (UID: \"5c63b358-0b01-466a-8bc6-88771b1a86bd\") " Oct 30 19:42:04.792321 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.792295 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c63b358-0b01-466a-8bc6-88771b1a86bd-maas-api-openshift-ca" pod "5c63b358-0b01-466a-8bc6-88771b1a86bd" (UID: "5c63b358-0b01-466a-8bc6-88771b1a86bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:04.792472 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.792447 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c63b358-0b01-466a-8bc6-88771b1a86bd-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:04.794244 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.794220 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c63b358-0b01-466a-8bc6-88771b1a86bd-maas-api-xks-ca" pod "5c63b358-0b01-466a-8bc6-88771b1a86bd" (UID: "5c63b358-0b01-466a-8bc6-88771b1a86bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:42:04.794328 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.794241 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c63b358-0b01-466a-8bc6-88771b1a86bd-kube-api-access-5swtz" pod "5c63b358-0b01-466a-8bc6-88771b1a86bd" (UID: "5c63b358-0b01-466a-8bc6-88771b1a86bd"). InnerVolumeSpecName "kube-api-access-5swtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:04.893195 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.893093 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5swtz\" (UniqueName: \"kubernetes.io/projected/5c63b358-0b01-466a-8bc6-88771b1a86bd-kube-api-access-5swtz\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:04.893195 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:04.893150 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c63b358-0b01-466a-8bc6-88771b1a86bd-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:05.552416 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:05.552390 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" Oct 30 19:42:05.552963 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:05.552435 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj" event={"ID":"678454ff-4761-4bd4-b2a3-2b431bca7934","Type":"ContainerDied","Data":"758b41ef548f12d5938c3d35cd43697258b8dcca0015e8a59d4e4e1701a8422e"} Oct 30 19:42:05.552963 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:05.552483 4531 scope.go:122] "RemoveContainer" containerID="e882a37b69b67c8744ed27e70afaf53f06c0d39d1bc7245cadd52111c312db60" Oct 30 19:42:05.553925 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:05.553908 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc" Oct 30 19:42:05.554050 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:05.553906 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc" event={"ID":"5c63b358-0b01-466a-8bc6-88771b1a86bd","Type":"ContainerDied","Data":"7fe96caedfb0984accdb4d6d3ca3a49f972e483ab7a2b4ce9e7cb93820656646"} Oct 30 19:42:05.554195 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:05.554075 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fe96caedfb0984accdb4d6d3ca3a49f972e483ab7a2b4ce9e7cb93820656646" Oct 30 19:42:05.769858 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:05.769819 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj"] Oct 30 19:42:05.784156 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:05.784109 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-4a7a52/simulator-w2-4a7a52-kserve-575579b57-nzxcj"] Oct 30 19:42:06.098622 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:06.098536 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="678454ff-4761-4bd4-b2a3-2b431bca7934" path="/var/lib/kubelet/pods/678454ff-4761-4bd4-b2a3-2b431bca7934/volumes" Oct 30 19:42:12.272105 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:12.272007 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v" podUID="06335665-17e4-4b6a-b7a6-30ef62625cfe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:42:12.859281 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:12.859244 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn"] Oct 30 19:42:12.977981 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:12.977938 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w"] Oct 30 19:42:17.860442 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:17.860368 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn" podUID="deda7597-7c2f-434d-83d0-a6f61c59cae9" containerName="payload-processing" containerID="cri-o://6eec38a8f73f5f2142786698238f62d9504f675787d34e3245de075783dabdb7" gracePeriod=25 Oct 30 19:42:17.979253 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:17.979188 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w" podUID="061d2407-1689-4ae4-b3c9-25d519671591" containerName="payload-pre-processing" containerID="cri-o://b6d2a173931f818d2525e05d7f54345897e61e19f5603e213dd0bfc4788be00a" gracePeriod=25 Oct 30 19:42:20.640614 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:20.640538 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v" Oct 30 19:42:20.641414 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:20.641393 4531 generic.go:356] "Generic (PLEG): container finished" podID="06335665-17e4-4b6a-b7a6-30ef62625cfe" containerID="4163e63fc110f331c0d6d095431820d6ae72f53e474a82e0df07ce10498e43a5" exitCode=0 Oct 30 19:42:20.641525 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:20.641468 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v" event={"ID":"06335665-17e4-4b6a-b7a6-30ef62625cfe","Type":"ContainerDied","Data":"4163e63fc110f331c0d6d095431820d6ae72f53e474a82e0df07ce10498e43a5"} Oct 30 19:42:20.641565 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:20.641530 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v" event={"ID":"06335665-17e4-4b6a-b7a6-30ef62625cfe","Type":"ContainerDied","Data":"e45aed8b09b8887278f151d3c4addfeafbee87affb77b64d56721ca0dddd69ff"} Oct 30 19:42:20.641565 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:20.641554 4531 scope.go:122] "RemoveContainer" containerID="4163e63fc110f331c0d6d095431820d6ae72f53e474a82e0df07ce10498e43a5" Oct 30 19:42:20.650841 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:20.650535 4531 scope.go:122] "RemoveContainer" containerID="4163e63fc110f331c0d6d095431820d6ae72f53e474a82e0df07ce10498e43a5" Oct 30 19:42:20.650841 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:42:20.650828 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4163e63fc110f331c0d6d095431820d6ae72f53e474a82e0df07ce10498e43a5\": container with ID starting with 4163e63fc110f331c0d6d095431820d6ae72f53e474a82e0df07ce10498e43a5 not found: ID does not exist" containerID="4163e63fc110f331c0d6d095431820d6ae72f53e474a82e0df07ce10498e43a5" Oct 30 19:42:20.651065 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:20.650857 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4163e63fc110f331c0d6d095431820d6ae72f53e474a82e0df07ce10498e43a5"} err="failed to get container status \"4163e63fc110f331c0d6d095431820d6ae72f53e474a82e0df07ce10498e43a5\": rpc error: code = NotFound desc = could not find container \"4163e63fc110f331c0d6d095431820d6ae72f53e474a82e0df07ce10498e43a5\": container with ID starting with 4163e63fc110f331c0d6d095431820d6ae72f53e474a82e0df07ce10498e43a5 not found: ID does not exist" Oct 30 19:42:20.658947 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:20.658917 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06335665-17e4-4b6a-b7a6-30ef62625cfe-kube-api-access-dx7wv\" (UniqueName: \"kubernetes.io/projected/06335665-17e4-4b6a-b7a6-30ef62625cfe-kube-api-access-dx7wv\") pod \"06335665-17e4-4b6a-b7a6-30ef62625cfe\" (UID: \"06335665-17e4-4b6a-b7a6-30ef62625cfe\") " Oct 30 19:42:20.659076 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:20.658976 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06335665-17e4-4b6a-b7a6-30ef62625cfe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06335665-17e4-4b6a-b7a6-30ef62625cfe-plugins-config-volume\") pod \"06335665-17e4-4b6a-b7a6-30ef62625cfe\" (UID: \"06335665-17e4-4b6a-b7a6-30ef62625cfe\") " Oct 30 19:42:20.659399 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:20.659379 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06335665-17e4-4b6a-b7a6-30ef62625cfe-plugins-config-volume" pod "06335665-17e4-4b6a-b7a6-30ef62625cfe" (UID: "06335665-17e4-4b6a-b7a6-30ef62625cfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:20.661170 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:20.661148 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06335665-17e4-4b6a-b7a6-30ef62625cfe-kube-api-access-dx7wv" pod "06335665-17e4-4b6a-b7a6-30ef62625cfe" (UID: "06335665-17e4-4b6a-b7a6-30ef62625cfe"). InnerVolumeSpecName "kube-api-access-dx7wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:20.760787 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:20.760748 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx7wv\" (UniqueName: \"kubernetes.io/projected/06335665-17e4-4b6a-b7a6-30ef62625cfe-kube-api-access-dx7wv\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:20.760787 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:20.760783 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06335665-17e4-4b6a-b7a6-30ef62625cfe-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:21.647698 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:21.647670 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v" Oct 30 19:42:21.812393 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:21.812349 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v"] Oct 30 19:42:21.860777 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:21.860745 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4a7a52-7d794f9446-x227v"] Oct 30 19:42:22.098516 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:22.098472 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06335665-17e4-4b6a-b7a6-30ef62625cfe" path="/var/lib/kubelet/pods/06335665-17e4-4b6a-b7a6-30ef62625cfe/volumes" Oct 30 19:42:24.064498 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:24.064462 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4"] Oct 30 19:42:24.113280 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:24.113248 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d4f84d-67df4"] Oct 30 19:42:25.150596 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:25.150560 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w" podUID="061d2407-1689-4ae4-b3c9-25d519671591" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:42:25.151006 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:25.150566 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn" podUID="deda7597-7c2f-434d-83d0-a6f61c59cae9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:42:26.099609 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:26.099570 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daa39ec7-9f92-411e-aa6f-209aa15b43a2" path="/var/lib/kubelet/pods/daa39ec7-9f92-411e-aa6f-209aa15b43a2/volumes" Oct 30 19:42:26.397549 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:42:26.397466 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:42:28.465089 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.465057 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea583398-jdbxh"] Oct 30 19:42:28.467256 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.465453 4531 cpu_manager.go:414] "containerMap: removing container" podUID="678454ff-4761-4bd4-b2a3-2b431bca7934" containerName="main" Oct 30 19:42:28.467256 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.465472 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="678454ff-4761-4bd4-b2a3-2b431bca7934" containerName="main" Oct 30 19:42:28.467256 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.465498 4531 cpu_manager.go:414] "containerMap: removing container" podUID="5c63b358-0b01-466a-8bc6-88771b1a86bd" containerName="revoke-keys" Oct 30 19:42:28.467256 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.465504 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c63b358-0b01-466a-8bc6-88771b1a86bd" containerName="revoke-keys" Oct 30 19:42:28.467256 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.465513 4531 cpu_manager.go:414] "containerMap: removing container" podUID="06335665-17e4-4b6a-b7a6-30ef62625cfe" containerName="payload-processing" Oct 30 19:42:28.467256 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.465518 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06335665-17e4-4b6a-b7a6-30ef62625cfe" containerName="payload-processing" Oct 30 19:42:28.467256 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.465580 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="678454ff-4761-4bd4-b2a3-2b431bca7934" containerName="main" Oct 30 19:42:28.467256 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.465593 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c63b358-0b01-466a-8bc6-88771b1a86bd" containerName="revoke-keys" Oct 30 19:42:28.467256 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.465599 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="06335665-17e4-4b6a-b7a6-30ef62625cfe" containerName="payload-processing" Oct 30 19:42:28.468176 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.468160 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea583398-jdbxh" Oct 30 19:42:28.522330 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.522301 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:42:28.524780 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.524762 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:42:28.540818 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.540782 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfe5743d-6d47-48a3-b86c-927895ba3caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea583398-jdbxh\" (UID: \"cfe5743d-6d47-48a3-b86c-927895ba3caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea583398-jdbxh" Oct 30 19:42:28.540818 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.540829 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfe5743d-6d47-48a3-b86c-927895ba3caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea583398-jdbxh\" (UID: \"cfe5743d-6d47-48a3-b86c-927895ba3caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea583398-jdbxh" Oct 30 19:42:28.541096 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.541079 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28wvd\" (UniqueName: \"kubernetes.io/projected/cfe5743d-6d47-48a3-b86c-927895ba3caf-kube-api-access-28wvd\") pod \"maas-api-revoke-sub-ea583398-jdbxh\" (UID: \"cfe5743d-6d47-48a3-b86c-927895ba3caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea583398-jdbxh" Oct 30 19:42:28.571783 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.571751 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:42:28.571783 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.571751 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:42:28.574197 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.574146 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c907a8b-5wvmk"] Oct 30 19:42:28.578767 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.578737 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea583398-jdbxh"] Oct 30 19:42:28.578927 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.578856 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c907a8b-5wvmk" Oct 30 19:42:28.592373 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.592341 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c907a8b-5wvmk"] Oct 30 19:42:28.642096 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.642061 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dkk7\" (UniqueName: \"kubernetes.io/projected/71a76f93-3d23-43e6-a3ac-c63938d30f53-kube-api-access-9dkk7\") pod \"maas-api-revoke-sub-5c907a8b-5wvmk\" (UID: \"71a76f93-3d23-43e6-a3ac-c63938d30f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c907a8b-5wvmk" Oct 30 19:42:28.642340 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.642158 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28wvd\" (UniqueName: \"kubernetes.io/projected/cfe5743d-6d47-48a3-b86c-927895ba3caf-kube-api-access-28wvd\") pod \"maas-api-revoke-sub-ea583398-jdbxh\" (UID: \"cfe5743d-6d47-48a3-b86c-927895ba3caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea583398-jdbxh" Oct 30 19:42:28.642340 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.642228 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfe5743d-6d47-48a3-b86c-927895ba3caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea583398-jdbxh\" (UID: \"cfe5743d-6d47-48a3-b86c-927895ba3caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea583398-jdbxh" Oct 30 19:42:28.642340 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.642268 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfe5743d-6d47-48a3-b86c-927895ba3caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea583398-jdbxh\" (UID: \"cfe5743d-6d47-48a3-b86c-927895ba3caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea583398-jdbxh" Oct 30 19:42:28.642340 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.642315 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a76f93-3d23-43e6-a3ac-c63938d30f53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c907a8b-5wvmk\" (UID: \"71a76f93-3d23-43e6-a3ac-c63938d30f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c907a8b-5wvmk" Oct 30 19:42:28.642616 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.642355 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a76f93-3d23-43e6-a3ac-c63938d30f53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c907a8b-5wvmk\" (UID: \"71a76f93-3d23-43e6-a3ac-c63938d30f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c907a8b-5wvmk" Oct 30 19:42:28.642969 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.642940 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfe5743d-6d47-48a3-b86c-927895ba3caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea583398-jdbxh\" (UID: \"cfe5743d-6d47-48a3-b86c-927895ba3caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea583398-jdbxh" Oct 30 19:42:28.644893 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.644871 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfe5743d-6d47-48a3-b86c-927895ba3caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea583398-jdbxh\" (UID: \"cfe5743d-6d47-48a3-b86c-927895ba3caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea583398-jdbxh" Oct 30 19:42:28.711332 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.711299 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28wvd\" (UniqueName: \"kubernetes.io/projected/cfe5743d-6d47-48a3-b86c-927895ba3caf-kube-api-access-28wvd\") pod \"maas-api-revoke-sub-ea583398-jdbxh\" (UID: \"cfe5743d-6d47-48a3-b86c-927895ba3caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea583398-jdbxh" Oct 30 19:42:28.743195 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.743163 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a76f93-3d23-43e6-a3ac-c63938d30f53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c907a8b-5wvmk\" (UID: \"71a76f93-3d23-43e6-a3ac-c63938d30f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c907a8b-5wvmk" Oct 30 19:42:28.743370 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.743208 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a76f93-3d23-43e6-a3ac-c63938d30f53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c907a8b-5wvmk\" (UID: \"71a76f93-3d23-43e6-a3ac-c63938d30f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c907a8b-5wvmk" Oct 30 19:42:28.743370 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.743269 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dkk7\" (UniqueName: \"kubernetes.io/projected/71a76f93-3d23-43e6-a3ac-c63938d30f53-kube-api-access-9dkk7\") pod \"maas-api-revoke-sub-5c907a8b-5wvmk\" (UID: \"71a76f93-3d23-43e6-a3ac-c63938d30f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c907a8b-5wvmk" Oct 30 19:42:28.743956 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.743927 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a76f93-3d23-43e6-a3ac-c63938d30f53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c907a8b-5wvmk\" (UID: \"71a76f93-3d23-43e6-a3ac-c63938d30f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c907a8b-5wvmk" Oct 30 19:42:28.745740 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.745717 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a76f93-3d23-43e6-a3ac-c63938d30f53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c907a8b-5wvmk\" (UID: \"71a76f93-3d23-43e6-a3ac-c63938d30f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c907a8b-5wvmk" Oct 30 19:42:28.781227 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.781190 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea583398-jdbxh" Oct 30 19:42:28.805201 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.805105 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dkk7\" (UniqueName: \"kubernetes.io/projected/71a76f93-3d23-43e6-a3ac-c63938d30f53-kube-api-access-9dkk7\") pod \"maas-api-revoke-sub-5c907a8b-5wvmk\" (UID: \"71a76f93-3d23-43e6-a3ac-c63938d30f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c907a8b-5wvmk" Oct 30 19:42:28.893147 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:28.893092 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c907a8b-5wvmk" Oct 30 19:42:29.162156 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:29.162111 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea583398-jdbxh"] Oct 30 19:42:29.163894 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:42:29.163867 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe5743d_6d47_48a3_b86c_927895ba3caf.slice/crio-59f73e73bbf26d7f035937c16dc700b39fe27b4d28b9dc07c648bc9a94b67ea1 WatchSource:0}: Error finding container 59f73e73bbf26d7f035937c16dc700b39fe27b4d28b9dc07c648bc9a94b67ea1: Status 404 returned error can't find the container with id 59f73e73bbf26d7f035937c16dc700b39fe27b4d28b9dc07c648bc9a94b67ea1 Oct 30 19:42:29.271178 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:29.271142 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c907a8b-5wvmk"] Oct 30 19:42:29.274242 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:42:29.274211 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a76f93_3d23_43e6_a3ac_c63938d30f53.slice/crio-d868704cd937f49cd5917764c49eaa84193fe189dfe3847e1728857c9ec38625 WatchSource:0}: Error finding container d868704cd937f49cd5917764c49eaa84193fe189dfe3847e1728857c9ec38625: Status 404 returned error can't find the container with id d868704cd937f49cd5917764c49eaa84193fe189dfe3847e1728857c9ec38625 Oct 30 19:42:29.694978 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:29.694875 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea583398-jdbxh" event={"ID":"cfe5743d-6d47-48a3-b86c-927895ba3caf","Type":"ContainerStarted","Data":"c281231f18e214850382ff18919125fbc150f07dd5de08801e6b9da01004a714"} Oct 30 19:42:29.694978 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:29.694944 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea583398-jdbxh" event={"ID":"cfe5743d-6d47-48a3-b86c-927895ba3caf","Type":"ContainerStarted","Data":"59f73e73bbf26d7f035937c16dc700b39fe27b4d28b9dc07c648bc9a94b67ea1"} Oct 30 19:42:29.696594 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:29.696566 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c907a8b-5wvmk" event={"ID":"71a76f93-3d23-43e6-a3ac-c63938d30f53","Type":"ContainerStarted","Data":"8035e4ac8cfdb2b6c18ede3317efa2df156cceaa4a0cc88b0bf97ba668cc5f7f"} Oct 30 19:42:29.696721 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:29.696615 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c907a8b-5wvmk" event={"ID":"71a76f93-3d23-43e6-a3ac-c63938d30f53","Type":"ContainerStarted","Data":"d868704cd937f49cd5917764c49eaa84193fe189dfe3847e1728857c9ec38625"} Oct 30 19:42:29.814864 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:29.814811 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea583398-jdbxh" podStartSLOduration=1.814797781 podStartE2EDuration="1.814797781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:29.813780978 +0000 UTC m=+4109.840885936" watchObservedRunningTime="2026-10-30 19:42:29.814797781 +0000 UTC m=+4109.841902740" Oct 30 19:42:29.934534 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:29.934478 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c907a8b-5wvmk" podStartSLOduration=1.934462932 podStartE2EDuration="1.934462932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:29.932437752 +0000 UTC m=+4109.959542723" watchObservedRunningTime="2026-10-30 19:42:29.934462932 +0000 UTC m=+4109.961567890" Oct 30 19:42:30.705180 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:30.705144 4531 generic.go:356] "Generic (PLEG): container finished" podID="cfe5743d-6d47-48a3-b86c-927895ba3caf" containerID="c281231f18e214850382ff18919125fbc150f07dd5de08801e6b9da01004a714" exitCode=0 Oct 30 19:42:30.705752 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:30.705240 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea583398-jdbxh" event={"ID":"cfe5743d-6d47-48a3-b86c-927895ba3caf","Type":"ContainerDied","Data":"c281231f18e214850382ff18919125fbc150f07dd5de08801e6b9da01004a714"} Oct 30 19:42:30.706983 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:30.706959 4531 generic.go:356] "Generic (PLEG): container finished" podID="71a76f93-3d23-43e6-a3ac-c63938d30f53" containerID="8035e4ac8cfdb2b6c18ede3317efa2df156cceaa4a0cc88b0bf97ba668cc5f7f" exitCode=0 Oct 30 19:42:30.707122 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:30.707030 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c907a8b-5wvmk" event={"ID":"71a76f93-3d23-43e6-a3ac-c63938d30f53","Type":"ContainerDied","Data":"8035e4ac8cfdb2b6c18ede3317efa2df156cceaa4a0cc88b0bf97ba668cc5f7f"} Oct 30 19:42:31.853547 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:31.853512 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c907a8b-5wvmk" Oct 30 19:42:31.884034 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:31.884008 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea583398-jdbxh" Oct 30 19:42:31.978596 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:31.978510 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfe5743d-6d47-48a3-b86c-927895ba3caf-kube-api-access-28wvd\" (UniqueName: \"kubernetes.io/projected/cfe5743d-6d47-48a3-b86c-927895ba3caf-kube-api-access-28wvd\") pod \"cfe5743d-6d47-48a3-b86c-927895ba3caf\" (UID: \"cfe5743d-6d47-48a3-b86c-927895ba3caf\") " Oct 30 19:42:31.978786 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:31.978635 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfe5743d-6d47-48a3-b86c-927895ba3caf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfe5743d-6d47-48a3-b86c-927895ba3caf-maas-api-openshift-ca\") pod \"cfe5743d-6d47-48a3-b86c-927895ba3caf\" (UID: \"cfe5743d-6d47-48a3-b86c-927895ba3caf\") " Oct 30 19:42:31.978786 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:31.978699 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71a76f93-3d23-43e6-a3ac-c63938d30f53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a76f93-3d23-43e6-a3ac-c63938d30f53-maas-api-openshift-ca\") pod \"71a76f93-3d23-43e6-a3ac-c63938d30f53\" (UID: \"71a76f93-3d23-43e6-a3ac-c63938d30f53\") " Oct 30 19:42:31.978786 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:31.978777 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71a76f93-3d23-43e6-a3ac-c63938d30f53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a76f93-3d23-43e6-a3ac-c63938d30f53-maas-api-xks-ca\") pod \"71a76f93-3d23-43e6-a3ac-c63938d30f53\" (UID: \"71a76f93-3d23-43e6-a3ac-c63938d30f53\") " Oct 30 19:42:31.978962 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:31.978857 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfe5743d-6d47-48a3-b86c-927895ba3caf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfe5743d-6d47-48a3-b86c-927895ba3caf-maas-api-xks-ca\") pod \"cfe5743d-6d47-48a3-b86c-927895ba3caf\" (UID: \"cfe5743d-6d47-48a3-b86c-927895ba3caf\") " Oct 30 19:42:31.978962 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:31.978895 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a76f93-3d23-43e6-a3ac-c63938d30f53-kube-api-access-9dkk7\" (UniqueName: \"kubernetes.io/projected/71a76f93-3d23-43e6-a3ac-c63938d30f53-kube-api-access-9dkk7\") pod \"71a76f93-3d23-43e6-a3ac-c63938d30f53\" (UID: \"71a76f93-3d23-43e6-a3ac-c63938d30f53\") " Oct 30 19:42:31.979077 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:31.978999 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe5743d-6d47-48a3-b86c-927895ba3caf-maas-api-openshift-ca" pod "cfe5743d-6d47-48a3-b86c-927895ba3caf" (UID: "cfe5743d-6d47-48a3-b86c-927895ba3caf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:31.979164 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:31.979073 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a76f93-3d23-43e6-a3ac-c63938d30f53-maas-api-openshift-ca" pod "71a76f93-3d23-43e6-a3ac-c63938d30f53" (UID: "71a76f93-3d23-43e6-a3ac-c63938d30f53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:31.979513 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:31.979487 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfe5743d-6d47-48a3-b86c-927895ba3caf-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:31.979776 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:31.979686 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71a76f93-3d23-43e6-a3ac-c63938d30f53-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:31.981089 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:31.981061 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe5743d-6d47-48a3-b86c-927895ba3caf-kube-api-access-28wvd" pod "cfe5743d-6d47-48a3-b86c-927895ba3caf" (UID: "cfe5743d-6d47-48a3-b86c-927895ba3caf"). InnerVolumeSpecName "kube-api-access-28wvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:31.981473 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:31.981442 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a76f93-3d23-43e6-a3ac-c63938d30f53-maas-api-xks-ca" pod "71a76f93-3d23-43e6-a3ac-c63938d30f53" (UID: "71a76f93-3d23-43e6-a3ac-c63938d30f53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:42:31.981551 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:31.981459 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a76f93-3d23-43e6-a3ac-c63938d30f53-kube-api-access-9dkk7" pod "71a76f93-3d23-43e6-a3ac-c63938d30f53" (UID: "71a76f93-3d23-43e6-a3ac-c63938d30f53"). InnerVolumeSpecName "kube-api-access-9dkk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:31.981697 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:31.981681 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfe5743d-6d47-48a3-b86c-927895ba3caf-maas-api-xks-ca" pod "cfe5743d-6d47-48a3-b86c-927895ba3caf" (UID: "cfe5743d-6d47-48a3-b86c-927895ba3caf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:42:32.081212 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:32.081169 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71a76f93-3d23-43e6-a3ac-c63938d30f53-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:32.081212 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:32.081207 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfe5743d-6d47-48a3-b86c-927895ba3caf-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:32.081443 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:32.081222 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dkk7\" (UniqueName: \"kubernetes.io/projected/71a76f93-3d23-43e6-a3ac-c63938d30f53-kube-api-access-9dkk7\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:32.081443 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:32.081236 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28wvd\" (UniqueName: \"kubernetes.io/projected/cfe5743d-6d47-48a3-b86c-927895ba3caf-kube-api-access-28wvd\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:32.220684 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:32.220644 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll"] Oct 30 19:42:32.221306 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:32.220918 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" podUID="0ea84ad8-53fd-4c3d-a32f-d202b3075ac1" containerName="istio-proxy" containerID="cri-o://d67301a4f643826413b0828b0c2b559bf816956e0d8ee6c9a9a105bee53a0744" gracePeriod=30 Oct 30 19:42:32.720347 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:32.720313 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea583398-jdbxh" Oct 30 19:42:32.720347 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:32.720330 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea583398-jdbxh" event={"ID":"cfe5743d-6d47-48a3-b86c-927895ba3caf","Type":"ContainerDied","Data":"59f73e73bbf26d7f035937c16dc700b39fe27b4d28b9dc07c648bc9a94b67ea1"} Oct 30 19:42:32.720584 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:32.720383 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59f73e73bbf26d7f035937c16dc700b39fe27b4d28b9dc07c648bc9a94b67ea1" Oct 30 19:42:32.721755 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:32.721728 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c907a8b-5wvmk" event={"ID":"71a76f93-3d23-43e6-a3ac-c63938d30f53","Type":"ContainerDied","Data":"d868704cd937f49cd5917764c49eaa84193fe189dfe3847e1728857c9ec38625"} Oct 30 19:42:32.721879 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:32.721758 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c907a8b-5wvmk" Oct 30 19:42:32.721879 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:32.721774 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d868704cd937f49cd5917764c49eaa84193fe189dfe3847e1728857c9ec38625" Oct 30 19:42:33.421616 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:33.421576 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6"] Oct 30 19:42:33.422036 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:33.421977 4531 cpu_manager.go:414] "containerMap: removing container" podUID="71a76f93-3d23-43e6-a3ac-c63938d30f53" containerName="revoke-keys" Oct 30 19:42:33.422036 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:33.421987 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71a76f93-3d23-43e6-a3ac-c63938d30f53" containerName="revoke-keys" Oct 30 19:42:33.422036 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:33.422012 4531 cpu_manager.go:414] "containerMap: removing container" podUID="cfe5743d-6d47-48a3-b86c-927895ba3caf" containerName="revoke-keys" Oct 30 19:42:33.422036 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:33.422018 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfe5743d-6d47-48a3-b86c-927895ba3caf" containerName="revoke-keys" Oct 30 19:42:33.422217 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:33.422079 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfe5743d-6d47-48a3-b86c-927895ba3caf" containerName="revoke-keys" Oct 30 19:42:33.422217 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:33.422093 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="71a76f93-3d23-43e6-a3ac-c63938d30f53" containerName="revoke-keys" Oct 30 19:42:33.425380 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:33.425361 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6" Oct 30 19:42:33.479003 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:33.478945 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:42:33.479291 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:33.479091 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:42:33.479291 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:33.479278 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:42:33.483216 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:33.483189 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:42:33.485732 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:33.485703 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6"] Oct 30 19:42:33.494712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:33.494682 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a8ea62-3c75-4ddb-b63f-dd880ea8dca1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6\" (UID: \"60a8ea62-3c75-4ddb-b63f-dd880ea8dca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6" Oct 30 19:42:33.494865 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:33.494813 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a8ea62-3c75-4ddb-b63f-dd880ea8dca1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6\" (UID: \"60a8ea62-3c75-4ddb-b63f-dd880ea8dca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6" Oct 30 19:42:33.494906 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:33.494872 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n8x2\" (UniqueName: \"kubernetes.io/projected/60a8ea62-3c75-4ddb-b63f-dd880ea8dca1-kube-api-access-4n8x2\") pod \"maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6\" (UID: \"60a8ea62-3c75-4ddb-b63f-dd880ea8dca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6" Oct 30 19:42:33.596171 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:33.596112 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a8ea62-3c75-4ddb-b63f-dd880ea8dca1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6\" (UID: \"60a8ea62-3c75-4ddb-b63f-dd880ea8dca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6" Oct 30 19:42:33.596363 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:33.596243 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a8ea62-3c75-4ddb-b63f-dd880ea8dca1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6\" (UID: \"60a8ea62-3c75-4ddb-b63f-dd880ea8dca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6" Oct 30 19:42:33.596363 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:33.596314 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n8x2\" (UniqueName: \"kubernetes.io/projected/60a8ea62-3c75-4ddb-b63f-dd880ea8dca1-kube-api-access-4n8x2\") pod \"maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6\" (UID: \"60a8ea62-3c75-4ddb-b63f-dd880ea8dca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6" Oct 30 19:42:33.596753 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:33.596730 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a8ea62-3c75-4ddb-b63f-dd880ea8dca1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6\" (UID: \"60a8ea62-3c75-4ddb-b63f-dd880ea8dca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6" Oct 30 19:42:33.598785 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:33.598762 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a8ea62-3c75-4ddb-b63f-dd880ea8dca1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6\" (UID: \"60a8ea62-3c75-4ddb-b63f-dd880ea8dca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6" Oct 30 19:42:33.664376 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:33.664345 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n8x2\" (UniqueName: \"kubernetes.io/projected/60a8ea62-3c75-4ddb-b63f-dd880ea8dca1-kube-api-access-4n8x2\") pod \"maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6\" (UID: \"60a8ea62-3c75-4ddb-b63f-dd880ea8dca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6" Oct 30 19:42:33.738203 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:33.738080 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6" Oct 30 19:42:34.117413 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:34.117374 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6"] Oct 30 19:42:34.119765 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:42:34.119734 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a8ea62_3c75_4ddb_b63f_dd880ea8dca1.slice/crio-b700fde0ab9727174e530a74f5868c801d1c49368222aa21737c30360291aa27 WatchSource:0}: Error finding container b700fde0ab9727174e530a74f5868c801d1c49368222aa21737c30360291aa27: Status 404 returned error can't find the container with id b700fde0ab9727174e530a74f5868c801d1c49368222aa21737c30360291aa27 Oct 30 19:42:34.735996 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:34.735915 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6" event={"ID":"60a8ea62-3c75-4ddb-b63f-dd880ea8dca1","Type":"ContainerStarted","Data":"744dfbb24ff0f2f5018dc4638af1b2d0fd9f659dda807cc672d6f58be7d38d62"} Oct 30 19:42:34.735996 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:34.735969 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6" event={"ID":"60a8ea62-3c75-4ddb-b63f-dd880ea8dca1","Type":"ContainerStarted","Data":"b700fde0ab9727174e530a74f5868c801d1c49368222aa21737c30360291aa27"} Oct 30 19:42:34.871861 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:34.871808 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6" podStartSLOduration=1.871794458 podStartE2EDuration="1.871794458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:34.869566395 +0000 UTC m=+4114.896671368" watchObservedRunningTime="2026-10-30 19:42:34.871794458 +0000 UTC m=+4114.898899457" Oct 30 19:42:35.151368 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:35.151327 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn" podUID="deda7597-7c2f-434d-83d0-a6f61c59cae9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:42:35.151572 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:35.151374 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w" podUID="061d2407-1689-4ae4-b3c9-25d519671591" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:42:35.748373 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:35.748339 4531 generic.go:356] "Generic (PLEG): container finished" podID="60a8ea62-3c75-4ddb-b63f-dd880ea8dca1" containerID="744dfbb24ff0f2f5018dc4638af1b2d0fd9f659dda807cc672d6f58be7d38d62" exitCode=0 Oct 30 19:42:35.748802 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:35.748386 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6" event={"ID":"60a8ea62-3c75-4ddb-b63f-dd880ea8dca1","Type":"ContainerDied","Data":"744dfbb24ff0f2f5018dc4638af1b2d0fd9f659dda807cc672d6f58be7d38d62"} Oct 30 19:42:36.012744 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:36.012657 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc"] Oct 30 19:42:36.017805 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:36.017779 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a56c7c-q6wlc"] Oct 30 19:42:36.099348 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:36.099314 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c63b358-0b01-466a-8bc6-88771b1a86bd" path="/var/lib/kubelet/pods/5c63b358-0b01-466a-8bc6-88771b1a86bd/volumes" Oct 30 19:42:36.904303 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:36.904275 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6" Oct 30 19:42:37.034007 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.033917 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60a8ea62-3c75-4ddb-b63f-dd880ea8dca1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a8ea62-3c75-4ddb-b63f-dd880ea8dca1-maas-api-openshift-ca\") pod \"60a8ea62-3c75-4ddb-b63f-dd880ea8dca1\" (UID: \"60a8ea62-3c75-4ddb-b63f-dd880ea8dca1\") " Oct 30 19:42:37.034007 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.033998 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60a8ea62-3c75-4ddb-b63f-dd880ea8dca1-kube-api-access-4n8x2\" (UniqueName: \"kubernetes.io/projected/60a8ea62-3c75-4ddb-b63f-dd880ea8dca1-kube-api-access-4n8x2\") pod \"60a8ea62-3c75-4ddb-b63f-dd880ea8dca1\" (UID: \"60a8ea62-3c75-4ddb-b63f-dd880ea8dca1\") " Oct 30 19:42:37.034281 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.034061 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60a8ea62-3c75-4ddb-b63f-dd880ea8dca1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a8ea62-3c75-4ddb-b63f-dd880ea8dca1-maas-api-xks-ca\") pod \"60a8ea62-3c75-4ddb-b63f-dd880ea8dca1\" (UID: \"60a8ea62-3c75-4ddb-b63f-dd880ea8dca1\") " Oct 30 19:42:37.034352 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.034318 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60a8ea62-3c75-4ddb-b63f-dd880ea8dca1-maas-api-openshift-ca" pod "60a8ea62-3c75-4ddb-b63f-dd880ea8dca1" (UID: "60a8ea62-3c75-4ddb-b63f-dd880ea8dca1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:37.034524 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.034497 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a8ea62-3c75-4ddb-b63f-dd880ea8dca1-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:37.036312 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.036279 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60a8ea62-3c75-4ddb-b63f-dd880ea8dca1-maas-api-xks-ca" pod "60a8ea62-3c75-4ddb-b63f-dd880ea8dca1" (UID: "60a8ea62-3c75-4ddb-b63f-dd880ea8dca1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:42:37.036426 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.036402 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a8ea62-3c75-4ddb-b63f-dd880ea8dca1-kube-api-access-4n8x2" pod "60a8ea62-3c75-4ddb-b63f-dd880ea8dca1" (UID: "60a8ea62-3c75-4ddb-b63f-dd880ea8dca1"). InnerVolumeSpecName "kube-api-access-4n8x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:37.135696 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.135646 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n8x2\" (UniqueName: \"kubernetes.io/projected/60a8ea62-3c75-4ddb-b63f-dd880ea8dca1-kube-api-access-4n8x2\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:37.135696 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.135685 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a8ea62-3c75-4ddb-b63f-dd880ea8dca1-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:37.474222 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.474190 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:42:37.539890 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.539837 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-token\" (UniqueName: \"kubernetes.io/projected/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-token\") pod \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " Oct 30 19:42:37.539890 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.539903 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-envoy\") pod \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " Oct 30 19:42:37.540221 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.539956 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-data\") pod \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " Oct 30 19:42:37.540221 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.539976 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-kube-api-access-7qkc6\" (UniqueName: \"kubernetes.io/projected/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-kube-api-access-7qkc6\") pod \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " Oct 30 19:42:37.540221 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.540008 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istiod-ca-cert\") pod \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " Oct 30 19:42:37.540221 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.540059 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-podinfo\") pod \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " Oct 30 19:42:37.540221 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.540100 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-credential-socket\") pod \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " Oct 30 19:42:37.540221 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.540190 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-workload-certs\") pod \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " Oct 30 19:42:37.540566 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.540238 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-workload-socket\") pod \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\" (UID: \"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1\") " Oct 30 19:42:37.540566 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.540504 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-credential-socket" pod "0ea84ad8-53fd-4c3d-a32f-d202b3075ac1" (UID: "0ea84ad8-53fd-4c3d-a32f-d202b3075ac1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:37.540566 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.540527 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istiod-ca-cert" pod "0ea84ad8-53fd-4c3d-a32f-d202b3075ac1" (UID: "0ea84ad8-53fd-4c3d-a32f-d202b3075ac1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:37.540763 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.540729 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-data" pod "0ea84ad8-53fd-4c3d-a32f-d202b3075ac1" (UID: "0ea84ad8-53fd-4c3d-a32f-d202b3075ac1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:37.540928 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.540903 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:37.541005 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.540777 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-workload-socket" pod "0ea84ad8-53fd-4c3d-a32f-d202b3075ac1" (UID: "0ea84ad8-53fd-4c3d-a32f-d202b3075ac1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:37.541005 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.540945 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-workload-certs" pod "0ea84ad8-53fd-4c3d-a32f-d202b3075ac1" (UID: "0ea84ad8-53fd-4c3d-a32f-d202b3075ac1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:37.541005 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.540959 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:37.542466 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.542441 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-envoy" pod "0ea84ad8-53fd-4c3d-a32f-d202b3075ac1" (UID: "0ea84ad8-53fd-4c3d-a32f-d202b3075ac1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:37.542466 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.542451 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-token" pod "0ea84ad8-53fd-4c3d-a32f-d202b3075ac1" (UID: "0ea84ad8-53fd-4c3d-a32f-d202b3075ac1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:37.542970 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.542954 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-kube-api-access-7qkc6" pod "0ea84ad8-53fd-4c3d-a32f-d202b3075ac1" (UID: "0ea84ad8-53fd-4c3d-a32f-d202b3075ac1"). InnerVolumeSpecName "kube-api-access-7qkc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:37.543018 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.542963 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-podinfo" pod "0ea84ad8-53fd-4c3d-a32f-d202b3075ac1" (UID: "0ea84ad8-53fd-4c3d-a32f-d202b3075ac1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:42:37.641924 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.641819 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:37.641924 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.641855 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:37.641924 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.641869 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:37.641924 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.641880 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:37.641924 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.641892 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:37.641924 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.641902 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:37.641924 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.641914 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qkc6\" (UniqueName: \"kubernetes.io/projected/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1-kube-api-access-7qkc6\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:37.761286 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.761259 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6" Oct 30 19:42:37.761511 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.761296 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6" event={"ID":"60a8ea62-3c75-4ddb-b63f-dd880ea8dca1","Type":"ContainerDied","Data":"b700fde0ab9727174e530a74f5868c801d1c49368222aa21737c30360291aa27"} Oct 30 19:42:37.761511 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.761345 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b700fde0ab9727174e530a74f5868c801d1c49368222aa21737c30360291aa27" Oct 30 19:42:37.763012 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.762978 4531 generic.go:356] "Generic (PLEG): container finished" podID="0ea84ad8-53fd-4c3d-a32f-d202b3075ac1" containerID="d67301a4f643826413b0828b0c2b559bf816956e0d8ee6c9a9a105bee53a0744" exitCode=0 Oct 30 19:42:37.763141 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.763022 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" event={"ID":"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1","Type":"ContainerDied","Data":"d67301a4f643826413b0828b0c2b559bf816956e0d8ee6c9a9a105bee53a0744"} Oct 30 19:42:37.763141 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.763033 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" Oct 30 19:42:37.763141 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.763063 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll" event={"ID":"0ea84ad8-53fd-4c3d-a32f-d202b3075ac1","Type":"ContainerDied","Data":"d3d597aaed9140cf90329ee52e7a31a1df0864e2f5ed28bf9d798ee52ac61424"} Oct 30 19:42:37.763141 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.763091 4531 scope.go:122] "RemoveContainer" containerID="d67301a4f643826413b0828b0c2b559bf816956e0d8ee6c9a9a105bee53a0744" Oct 30 19:42:37.773863 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.773840 4531 scope.go:122] "RemoveContainer" containerID="9c555055604859de611acb6065076055f6996b57cb761f84f453b52621ee92be" Oct 30 19:42:37.784203 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.784178 4531 scope.go:122] "RemoveContainer" containerID="d67301a4f643826413b0828b0c2b559bf816956e0d8ee6c9a9a105bee53a0744" Oct 30 19:42:37.784559 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:42:37.784535 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67301a4f643826413b0828b0c2b559bf816956e0d8ee6c9a9a105bee53a0744\": container with ID starting with d67301a4f643826413b0828b0c2b559bf816956e0d8ee6c9a9a105bee53a0744 not found: ID does not exist" containerID="d67301a4f643826413b0828b0c2b559bf816956e0d8ee6c9a9a105bee53a0744" Oct 30 19:42:37.784619 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.784568 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67301a4f643826413b0828b0c2b559bf816956e0d8ee6c9a9a105bee53a0744"} err="failed to get container status \"d67301a4f643826413b0828b0c2b559bf816956e0d8ee6c9a9a105bee53a0744\": rpc error: code = NotFound desc = could not find container \"d67301a4f643826413b0828b0c2b559bf816956e0d8ee6c9a9a105bee53a0744\": container with ID starting with d67301a4f643826413b0828b0c2b559bf816956e0d8ee6c9a9a105bee53a0744 not found: ID does not exist" Oct 30 19:42:37.784619 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.784589 4531 scope.go:122] "RemoveContainer" containerID="9c555055604859de611acb6065076055f6996b57cb761f84f453b52621ee92be" Oct 30 19:42:37.784830 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:42:37.784814 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c555055604859de611acb6065076055f6996b57cb761f84f453b52621ee92be\": container with ID starting with 9c555055604859de611acb6065076055f6996b57cb761f84f453b52621ee92be not found: ID does not exist" containerID="9c555055604859de611acb6065076055f6996b57cb761f84f453b52621ee92be" Oct 30 19:42:37.784884 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.784830 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c555055604859de611acb6065076055f6996b57cb761f84f453b52621ee92be"} err="failed to get container status \"9c555055604859de611acb6065076055f6996b57cb761f84f453b52621ee92be\": rpc error: code = NotFound desc = could not find container \"9c555055604859de611acb6065076055f6996b57cb761f84f453b52621ee92be\": container with ID starting with 9c555055604859de611acb6065076055f6996b57cb761f84f453b52621ee92be not found: ID does not exist" Oct 30 19:42:37.817080 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.817048 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx"] Oct 30 19:42:37.837606 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.837574 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8"] Oct 30 19:42:37.845314 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:37.845282 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97"] Oct 30 19:42:38.115165 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.115095 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn" Oct 30 19:42:38.149185 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.149120 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll"] Oct 30 19:42:38.156104 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.156073 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d4f84d-openshift-default-689b8c4b99-bmzll"] Oct 30 19:42:38.242352 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.242324 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w" Oct 30 19:42:38.247363 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.247334 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deda7597-7c2f-434d-83d0-a6f61c59cae9-kube-api-access-24bw7\" (UniqueName: \"kubernetes.io/projected/deda7597-7c2f-434d-83d0-a6f61c59cae9-kube-api-access-24bw7\") pod \"deda7597-7c2f-434d-83d0-a6f61c59cae9\" (UID: \"deda7597-7c2f-434d-83d0-a6f61c59cae9\") " Oct 30 19:42:38.247491 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.247475 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deda7597-7c2f-434d-83d0-a6f61c59cae9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deda7597-7c2f-434d-83d0-a6f61c59cae9-plugins-config-volume\") pod \"deda7597-7c2f-434d-83d0-a6f61c59cae9\" (UID: \"deda7597-7c2f-434d-83d0-a6f61c59cae9\") " Oct 30 19:42:38.247933 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.247904 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deda7597-7c2f-434d-83d0-a6f61c59cae9-plugins-config-volume" pod "deda7597-7c2f-434d-83d0-a6f61c59cae9" (UID: "deda7597-7c2f-434d-83d0-a6f61c59cae9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:38.250092 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.250059 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deda7597-7c2f-434d-83d0-a6f61c59cae9-kube-api-access-24bw7" pod "deda7597-7c2f-434d-83d0-a6f61c59cae9" (UID: "deda7597-7c2f-434d-83d0-a6f61c59cae9"). InnerVolumeSpecName "kube-api-access-24bw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:38.348772 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.348729 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/061d2407-1689-4ae4-b3c9-25d519671591-kube-api-access-xxm68\" (UniqueName: \"kubernetes.io/projected/061d2407-1689-4ae4-b3c9-25d519671591-kube-api-access-xxm68\") pod \"061d2407-1689-4ae4-b3c9-25d519671591\" (UID: \"061d2407-1689-4ae4-b3c9-25d519671591\") " Oct 30 19:42:38.348985 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.348824 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/061d2407-1689-4ae4-b3c9-25d519671591-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/061d2407-1689-4ae4-b3c9-25d519671591-plugins-config-volume\") pod \"061d2407-1689-4ae4-b3c9-25d519671591\" (UID: \"061d2407-1689-4ae4-b3c9-25d519671591\") " Oct 30 19:42:38.349240 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.349214 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/061d2407-1689-4ae4-b3c9-25d519671591-plugins-config-volume" pod "061d2407-1689-4ae4-b3c9-25d519671591" (UID: "061d2407-1689-4ae4-b3c9-25d519671591"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:38.349380 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.349238 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/deda7597-7c2f-434d-83d0-a6f61c59cae9-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:38.349380 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.349265 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24bw7\" (UniqueName: \"kubernetes.io/projected/deda7597-7c2f-434d-83d0-a6f61c59cae9-kube-api-access-24bw7\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:38.351109 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.351083 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061d2407-1689-4ae4-b3c9-25d519671591-kube-api-access-xxm68" pod "061d2407-1689-4ae4-b3c9-25d519671591" (UID: "061d2407-1689-4ae4-b3c9-25d519671591"). InnerVolumeSpecName "kube-api-access-xxm68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:38.449990 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.449946 4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/061d2407-1689-4ae4-b3c9-25d519671591-plugins-config-volume\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:38.449990 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.449979 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxm68\" (UniqueName: \"kubernetes.io/projected/061d2407-1689-4ae4-b3c9-25d519671591-kube-api-access-xxm68\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:38.771606 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.771568 4531 generic.go:356] "Generic (PLEG): container finished" podID="061d2407-1689-4ae4-b3c9-25d519671591" containerID="b6d2a173931f818d2525e05d7f54345897e61e19f5603e213dd0bfc4788be00a" exitCode=0 Oct 30 19:42:38.771795 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.771622 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w" Oct 30 19:42:38.771795 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.771663 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w" event={"ID":"061d2407-1689-4ae4-b3c9-25d519671591","Type":"ContainerDied","Data":"b6d2a173931f818d2525e05d7f54345897e61e19f5603e213dd0bfc4788be00a"} Oct 30 19:42:38.771795 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.771736 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w" event={"ID":"061d2407-1689-4ae4-b3c9-25d519671591","Type":"ContainerDied","Data":"4a0abd089f3bb90dc306baf43ef89881d180aa55c30f118bf7c3a3a257563f1c"} Oct 30 19:42:38.771795 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.771765 4531 scope.go:122] "RemoveContainer" containerID="b6d2a173931f818d2525e05d7f54345897e61e19f5603e213dd0bfc4788be00a" Oct 30 19:42:38.773245 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.773216 4531 generic.go:356] "Generic (PLEG): container finished" podID="deda7597-7c2f-434d-83d0-a6f61c59cae9" containerID="6eec38a8f73f5f2142786698238f62d9504f675787d34e3245de075783dabdb7" exitCode=0 Oct 30 19:42:38.773378 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.773271 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn" Oct 30 19:42:38.773378 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.773332 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn" event={"ID":"deda7597-7c2f-434d-83d0-a6f61c59cae9","Type":"ContainerDied","Data":"6eec38a8f73f5f2142786698238f62d9504f675787d34e3245de075783dabdb7"} Oct 30 19:42:38.773586 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.773543 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn" event={"ID":"deda7597-7c2f-434d-83d0-a6f61c59cae9","Type":"ContainerDied","Data":"5164948a3bf2400d16a0f68844c481d873c9644abadab44e2a3bdcff86125d41"} Oct 30 19:42:38.782827 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.782799 4531 scope.go:122] "RemoveContainer" containerID="b6d2a173931f818d2525e05d7f54345897e61e19f5603e213dd0bfc4788be00a" Oct 30 19:42:38.783221 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:42:38.783105 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6d2a173931f818d2525e05d7f54345897e61e19f5603e213dd0bfc4788be00a\": container with ID starting with b6d2a173931f818d2525e05d7f54345897e61e19f5603e213dd0bfc4788be00a not found: ID does not exist" containerID="b6d2a173931f818d2525e05d7f54345897e61e19f5603e213dd0bfc4788be00a" Oct 30 19:42:38.783221 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.783177 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6d2a173931f818d2525e05d7f54345897e61e19f5603e213dd0bfc4788be00a"} err="failed to get container status \"b6d2a173931f818d2525e05d7f54345897e61e19f5603e213dd0bfc4788be00a\": rpc error: code = NotFound desc = could not find container \"b6d2a173931f818d2525e05d7f54345897e61e19f5603e213dd0bfc4788be00a\": container with ID starting with b6d2a173931f818d2525e05d7f54345897e61e19f5603e213dd0bfc4788be00a not found: ID does not exist" Oct 30 19:42:38.783221 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.783202 4531 scope.go:122] "RemoveContainer" containerID="6eec38a8f73f5f2142786698238f62d9504f675787d34e3245de075783dabdb7" Oct 30 19:42:38.792978 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.792954 4531 scope.go:122] "RemoveContainer" containerID="6eec38a8f73f5f2142786698238f62d9504f675787d34e3245de075783dabdb7" Oct 30 19:42:38.793448 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:42:38.793419 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eec38a8f73f5f2142786698238f62d9504f675787d34e3245de075783dabdb7\": container with ID starting with 6eec38a8f73f5f2142786698238f62d9504f675787d34e3245de075783dabdb7 not found: ID does not exist" containerID="6eec38a8f73f5f2142786698238f62d9504f675787d34e3245de075783dabdb7" Oct 30 19:42:38.793542 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.793471 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eec38a8f73f5f2142786698238f62d9504f675787d34e3245de075783dabdb7"} err="failed to get container status \"6eec38a8f73f5f2142786698238f62d9504f675787d34e3245de075783dabdb7\": rpc error: code = NotFound desc = could not find container \"6eec38a8f73f5f2142786698238f62d9504f675787d34e3245de075783dabdb7\": container with ID starting with 6eec38a8f73f5f2142786698238f62d9504f675787d34e3245de075783dabdb7 not found: ID does not exist" Oct 30 19:42:38.936617 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.936584 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn"] Oct 30 19:42:38.942673 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:38.942644 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a56c7c-978dff5bd-q6hpn"] Oct 30 19:42:39.051794 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:39.051710 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w"] Oct 30 19:42:39.057184 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:39.057153 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a56c7c-77994466f9-r2m8w"] Oct 30 19:42:40.097612 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:40.097569 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="061d2407-1689-4ae4-b3c9-25d519671591" path="/var/lib/kubelet/pods/061d2407-1689-4ae4-b3c9-25d519671591/volumes" Oct 30 19:42:40.098262 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:40.098223 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ea84ad8-53fd-4c3d-a32f-d202b3075ac1" path="/var/lib/kubelet/pods/0ea84ad8-53fd-4c3d-a32f-d202b3075ac1/volumes" Oct 30 19:42:40.098670 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:40.098648 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deda7597-7c2f-434d-83d0-a6f61c59cae9" path="/var/lib/kubelet/pods/deda7597-7c2f-434d-83d0-a6f61c59cae9/volumes" Oct 30 19:42:45.496174 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.496122 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7"] Oct 30 19:42:45.496688 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.496669 4531 cpu_manager.go:414] "containerMap: removing container" podUID="061d2407-1689-4ae4-b3c9-25d519671591" containerName="payload-pre-processing" Oct 30 19:42:45.496737 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.496693 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="061d2407-1689-4ae4-b3c9-25d519671591" containerName="payload-pre-processing" Oct 30 19:42:45.496776 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.496736 4531 cpu_manager.go:414] "containerMap: removing container" podUID="deda7597-7c2f-434d-83d0-a6f61c59cae9" containerName="payload-processing" Oct 30 19:42:45.496776 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.496745 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deda7597-7c2f-434d-83d0-a6f61c59cae9" containerName="payload-processing" Oct 30 19:42:45.496776 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.496759 4531 cpu_manager.go:414] "containerMap: removing container" podUID="0ea84ad8-53fd-4c3d-a32f-d202b3075ac1" containerName="istio-proxy" Oct 30 19:42:45.496776 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.496769 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ea84ad8-53fd-4c3d-a32f-d202b3075ac1" containerName="istio-proxy" Oct 30 19:42:45.496912 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.496796 4531 cpu_manager.go:414] "containerMap: removing container" podUID="60a8ea62-3c75-4ddb-b63f-dd880ea8dca1" containerName="revoke-keys" Oct 30 19:42:45.496912 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.496804 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60a8ea62-3c75-4ddb-b63f-dd880ea8dca1" containerName="revoke-keys" Oct 30 19:42:45.496912 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.496892 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="deda7597-7c2f-434d-83d0-a6f61c59cae9" containerName="payload-processing" Oct 30 19:42:45.496912 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.496908 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ea84ad8-53fd-4c3d-a32f-d202b3075ac1" containerName="istio-proxy" Oct 30 19:42:45.497037 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.496920 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="60a8ea62-3c75-4ddb-b63f-dd880ea8dca1" containerName="revoke-keys" Oct 30 19:42:45.497037 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.496930 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ea84ad8-53fd-4c3d-a32f-d202b3075ac1" containerName="istio-proxy" Oct 30 19:42:45.497037 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.496940 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="061d2407-1689-4ae4-b3c9-25d519671591" containerName="payload-pre-processing" Oct 30 19:42:45.501962 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.501938 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7" Oct 30 19:42:45.560280 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.559935 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7"] Oct 30 19:42:45.606404 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.606370 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:42:45.606598 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.606428 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:42:45.606598 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.606454 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:42:45.606598 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.606523 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:42:45.623116 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.623076 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r7qw\" (UniqueName: \"kubernetes.io/projected/b3ad1db0-bade-4eb9-9033-abdee1536bde-kube-api-access-7r7qw\") pod \"maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7\" (UID: \"b3ad1db0-bade-4eb9-9033-abdee1536bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7" Oct 30 19:42:45.623116 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.623116 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3ad1db0-bade-4eb9-9033-abdee1536bde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7\" (UID: \"b3ad1db0-bade-4eb9-9033-abdee1536bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7" Oct 30 19:42:45.623415 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.623295 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3ad1db0-bade-4eb9-9033-abdee1536bde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7\" (UID: \"b3ad1db0-bade-4eb9-9033-abdee1536bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7" Oct 30 19:42:45.724934 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.724895 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r7qw\" (UniqueName: \"kubernetes.io/projected/b3ad1db0-bade-4eb9-9033-abdee1536bde-kube-api-access-7r7qw\") pod \"maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7\" (UID: \"b3ad1db0-bade-4eb9-9033-abdee1536bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7" Oct 30 19:42:45.724934 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.724935 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3ad1db0-bade-4eb9-9033-abdee1536bde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7\" (UID: \"b3ad1db0-bade-4eb9-9033-abdee1536bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7" Oct 30 19:42:45.725227 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.724965 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3ad1db0-bade-4eb9-9033-abdee1536bde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7\" (UID: \"b3ad1db0-bade-4eb9-9033-abdee1536bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7" Oct 30 19:42:45.725770 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.725747 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3ad1db0-bade-4eb9-9033-abdee1536bde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7\" (UID: \"b3ad1db0-bade-4eb9-9033-abdee1536bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7" Oct 30 19:42:45.727588 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.727569 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3ad1db0-bade-4eb9-9033-abdee1536bde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7\" (UID: \"b3ad1db0-bade-4eb9-9033-abdee1536bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7" Oct 30 19:42:45.786440 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.786355 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r7qw\" (UniqueName: \"kubernetes.io/projected/b3ad1db0-bade-4eb9-9033-abdee1536bde-kube-api-access-7r7qw\") pod \"maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7\" (UID: \"b3ad1db0-bade-4eb9-9033-abdee1536bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7" Oct 30 19:42:45.815193 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:45.815154 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7" Oct 30 19:42:46.218955 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:46.218926 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7"] Oct 30 19:42:46.220501 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:42:46.220470 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ad1db0_bade_4eb9_9033_abdee1536bde.slice/crio-309fbf57169981d27f1d008e65336dcbbaaf0cffeedf346bc720273643b8c9fb WatchSource:0}: Error finding container 309fbf57169981d27f1d008e65336dcbbaaf0cffeedf346bc720273643b8c9fb: Status 404 returned error can't find the container with id 309fbf57169981d27f1d008e65336dcbbaaf0cffeedf346bc720273643b8c9fb Oct 30 19:42:46.829265 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:46.829180 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7" event={"ID":"b3ad1db0-bade-4eb9-9033-abdee1536bde","Type":"ContainerStarted","Data":"75aa2855a0145054251d2e361c4ec2637d95d2e0f031ab0139cf26ca6095ec44"} Oct 30 19:42:46.829265 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:46.829229 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7" event={"ID":"b3ad1db0-bade-4eb9-9033-abdee1536bde","Type":"ContainerStarted","Data":"309fbf57169981d27f1d008e65336dcbbaaf0cffeedf346bc720273643b8c9fb"} Oct 30 19:42:46.952316 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:46.952261 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7" podStartSLOduration=1.952244004 podStartE2EDuration="1.952244004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:46.948487157 +0000 UTC m=+4126.975592116" watchObservedRunningTime="2026-10-30 19:42:46.952244004 +0000 UTC m=+4126.979349018" Oct 30 19:42:47.837258 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:47.837215 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7" event={"ID":"b3ad1db0-bade-4eb9-9033-abdee1536bde","Type":"ContainerDied","Data":"75aa2855a0145054251d2e361c4ec2637d95d2e0f031ab0139cf26ca6095ec44"} Oct 30 19:42:47.837693 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:47.837123 4531 generic.go:356] "Generic (PLEG): container finished" podID="b3ad1db0-bade-4eb9-9033-abdee1536bde" containerID="75aa2855a0145054251d2e361c4ec2637d95d2e0f031ab0139cf26ca6095ec44" exitCode=0 Oct 30 19:42:48.972246 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:48.972217 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7" Oct 30 19:42:49.059287 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:49.059246 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ad1db0-bade-4eb9-9033-abdee1536bde-kube-api-access-7r7qw\" (UniqueName: \"kubernetes.io/projected/b3ad1db0-bade-4eb9-9033-abdee1536bde-kube-api-access-7r7qw\") pod \"b3ad1db0-bade-4eb9-9033-abdee1536bde\" (UID: \"b3ad1db0-bade-4eb9-9033-abdee1536bde\") " Oct 30 19:42:49.059495 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:49.059298 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3ad1db0-bade-4eb9-9033-abdee1536bde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3ad1db0-bade-4eb9-9033-abdee1536bde-maas-api-openshift-ca\") pod \"b3ad1db0-bade-4eb9-9033-abdee1536bde\" (UID: \"b3ad1db0-bade-4eb9-9033-abdee1536bde\") " Oct 30 19:42:49.059495 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:49.059346 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3ad1db0-bade-4eb9-9033-abdee1536bde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3ad1db0-bade-4eb9-9033-abdee1536bde-maas-api-xks-ca\") pod \"b3ad1db0-bade-4eb9-9033-abdee1536bde\" (UID: \"b3ad1db0-bade-4eb9-9033-abdee1536bde\") " Oct 30 19:42:49.059763 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:49.059732 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ad1db0-bade-4eb9-9033-abdee1536bde-maas-api-openshift-ca" pod "b3ad1db0-bade-4eb9-9033-abdee1536bde" (UID: "b3ad1db0-bade-4eb9-9033-abdee1536bde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:49.061694 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:49.061669 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3ad1db0-bade-4eb9-9033-abdee1536bde-maas-api-xks-ca" pod "b3ad1db0-bade-4eb9-9033-abdee1536bde" (UID: "b3ad1db0-bade-4eb9-9033-abdee1536bde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:42:49.061790 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:49.061747 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ad1db0-bade-4eb9-9033-abdee1536bde-kube-api-access-7r7qw" pod "b3ad1db0-bade-4eb9-9033-abdee1536bde" (UID: "b3ad1db0-bade-4eb9-9033-abdee1536bde"). InnerVolumeSpecName "kube-api-access-7r7qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:49.161052 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:49.160968 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3ad1db0-bade-4eb9-9033-abdee1536bde-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:49.161052 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:49.161001 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r7qw\" (UniqueName: \"kubernetes.io/projected/b3ad1db0-bade-4eb9-9033-abdee1536bde-kube-api-access-7r7qw\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:49.161052 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:49.161011 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3ad1db0-bade-4eb9-9033-abdee1536bde-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:49.793494 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:49.793449 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn"] Oct 30 19:42:49.842612 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:49.842574 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx"] Oct 30 19:42:49.844809 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:49.844772 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z"] Oct 30 19:42:49.847406 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:49.847376 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk"] Oct 30 19:42:49.854021 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:49.852183 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7" event={"ID":"b3ad1db0-bade-4eb9-9033-abdee1536bde","Type":"ContainerDied","Data":"309fbf57169981d27f1d008e65336dcbbaaf0cffeedf346bc720273643b8c9fb"} Oct 30 19:42:49.854021 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:49.852294 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="309fbf57169981d27f1d008e65336dcbbaaf0cffeedf346bc720273643b8c9fb" Oct 30 19:42:49.854021 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:49.852373 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7" Oct 30 19:42:52.839451 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:52.839408 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" podUID="f23b88e6-d3b4-4975-818a-22cee9d30dce" containerName="main" containerID="cri-o://8350fe722af29da10c0821946defc1c6e67f386563499b68f05fa2386ec5aee8" gracePeriod=45 Oct 30 19:42:52.861597 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:52.861558 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" podUID="0679a60b-f854-49dd-a87d-3ea640cac995" containerName="main" containerID="cri-o://cc78196bd0fb9c1faa36e9bbf14fe4cde15e8a49b6b1fd45406222a53368427a" gracePeriod=45 Oct 30 19:42:52.866560 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:52.866519 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" podUID="1a5f18bf-96ba-4edf-a758-206a1bd926a3" containerName="main" containerID="cri-o://9190ef4458cfdbc9586184bedbf49fe9c73e01228e2c297f4ba75f4b0d21fee4" gracePeriod=45 Oct 30 19:42:52.891775 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:52.891745 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" podUID="f23b88e6-d3b4-4975-818a-22cee9d30dce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": dial tcp 10.133.0.52:8000: connect: connection refused" Oct 30 19:42:52.914712 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:52.914672 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" podUID="1a5f18bf-96ba-4edf-a758-206a1bd926a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": dial tcp 10.133.0.40:8000: connect: connection refused" Oct 30 19:42:53.123747 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.123723 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 19:42:53.172911 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.172887 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 19:42:53.175859 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.175838 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 19:42:53.201539 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.201508 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-model-cache\") pod \"f23b88e6-d3b4-4975-818a-22cee9d30dce\" (UID: \"f23b88e6-d3b4-4975-818a-22cee9d30dce\") " Oct 30 19:42:53.201671 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.201575 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0679a60b-f854-49dd-a87d-3ea640cac995-kube-api-access-5pf9l\" (UniqueName: \"kubernetes.io/projected/0679a60b-f854-49dd-a87d-3ea640cac995-kube-api-access-5pf9l\") pod \"0679a60b-f854-49dd-a87d-3ea640cac995\" (UID: \"0679a60b-f854-49dd-a87d-3ea640cac995\") " Oct 30 19:42:53.201671 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.201610 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-model-cache\") pod \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\" (UID: \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\") " Oct 30 19:42:53.201671 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.201653 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-home\" (UniqueName: \"kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-home\") pod \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\" (UID: \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\") " Oct 30 19:42:53.201866 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.201684 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a5f18bf-96ba-4edf-a758-206a1bd926a3-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a5f18bf-96ba-4edf-a758-206a1bd926a3-tls-certs\") pod \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\" (UID: \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\") " Oct 30 19:42:53.201866 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.201734 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-model-cache\") pod \"0679a60b-f854-49dd-a87d-3ea640cac995\" (UID: \"0679a60b-f854-49dd-a87d-3ea640cac995\") " Oct 30 19:42:53.201866 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.201748 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-model-cache" pod "f23b88e6-d3b4-4975-818a-22cee9d30dce" (UID: "f23b88e6-d3b4-4975-818a-22cee9d30dce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:53.201866 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.201765 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f23b88e6-d3b4-4975-818a-22cee9d30dce-kube-api-access-t2rvw\" (UniqueName: \"kubernetes.io/projected/f23b88e6-d3b4-4975-818a-22cee9d30dce-kube-api-access-t2rvw\") pod \"f23b88e6-d3b4-4975-818a-22cee9d30dce\" (UID: \"f23b88e6-d3b4-4975-818a-22cee9d30dce\") " Oct 30 19:42:53.201866 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.201827 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-tmp-dir\") pod \"0679a60b-f854-49dd-a87d-3ea640cac995\" (UID: \"0679a60b-f854-49dd-a87d-3ea640cac995\") " Oct 30 19:42:53.202337 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.201865 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-model-cache" pod "1a5f18bf-96ba-4edf-a758-206a1bd926a3" (UID: "1a5f18bf-96ba-4edf-a758-206a1bd926a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:53.202337 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.201872 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-home\" (UniqueName: \"kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-home\") pod \"f23b88e6-d3b4-4975-818a-22cee9d30dce\" (UID: \"f23b88e6-d3b4-4975-818a-22cee9d30dce\") " Oct 30 19:42:53.202337 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.201973 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-dshm\") pod \"f23b88e6-d3b4-4975-818a-22cee9d30dce\" (UID: \"f23b88e6-d3b4-4975-818a-22cee9d30dce\") " Oct 30 19:42:53.202337 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.202021 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-dshm\") pod \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\" (UID: \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\") " Oct 30 19:42:53.202337 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.202026 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-model-cache" pod "0679a60b-f854-49dd-a87d-3ea640cac995" (UID: "0679a60b-f854-49dd-a87d-3ea640cac995"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:53.202337 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.202056 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-tmp-dir\") pod \"f23b88e6-d3b4-4975-818a-22cee9d30dce\" (UID: \"f23b88e6-d3b4-4975-818a-22cee9d30dce\") " Oct 30 19:42:53.202337 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.202066 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-home" pod "f23b88e6-d3b4-4975-818a-22cee9d30dce" (UID: "f23b88e6-d3b4-4975-818a-22cee9d30dce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:53.202337 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.202089 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-tmp-dir\") pod \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\" (UID: \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\") " Oct 30 19:42:53.202337 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.202120 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a5f18bf-96ba-4edf-a758-206a1bd926a3-kube-api-access-65r8x\" (UniqueName: \"kubernetes.io/projected/1a5f18bf-96ba-4edf-a758-206a1bd926a3-kube-api-access-65r8x\") pod \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\" (UID: \"1a5f18bf-96ba-4edf-a758-206a1bd926a3\") " Oct 30 19:42:53.202337 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.202208 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-dshm\") pod \"0679a60b-f854-49dd-a87d-3ea640cac995\" (UID: \"0679a60b-f854-49dd-a87d-3ea640cac995\") " Oct 30 19:42:53.202337 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.202242 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0679a60b-f854-49dd-a87d-3ea640cac995-tls-certs\" (UniqueName: \"kubernetes.io/secret/0679a60b-f854-49dd-a87d-3ea640cac995-tls-certs\") pod \"0679a60b-f854-49dd-a87d-3ea640cac995\" (UID: \"0679a60b-f854-49dd-a87d-3ea640cac995\") " Oct 30 19:42:53.202337 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.202282 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-home\" (UniqueName: \"kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-home\") pod \"0679a60b-f854-49dd-a87d-3ea640cac995\" (UID: \"0679a60b-f854-49dd-a87d-3ea640cac995\") " Oct 30 19:42:53.202337 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.202309 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f23b88e6-d3b4-4975-818a-22cee9d30dce-tls-certs\" (UniqueName: \"kubernetes.io/secret/f23b88e6-d3b4-4975-818a-22cee9d30dce-tls-certs\") pod \"f23b88e6-d3b4-4975-818a-22cee9d30dce\" (UID: \"f23b88e6-d3b4-4975-818a-22cee9d30dce\") " Oct 30 19:42:53.203360 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.202698 4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-model-cache\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:53.203360 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.202718 4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-home\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:53.203360 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.202732 4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-model-cache\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:53.203360 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.202206 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-tmp-dir" pod "0679a60b-f854-49dd-a87d-3ea640cac995" (UID: "0679a60b-f854-49dd-a87d-3ea640cac995"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:53.203360 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.202388 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-tmp-dir" pod "1a5f18bf-96ba-4edf-a758-206a1bd926a3" (UID: "1a5f18bf-96ba-4edf-a758-206a1bd926a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:53.203360 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.202723 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-tmp-dir" pod "f23b88e6-d3b4-4975-818a-22cee9d30dce" (UID: "f23b88e6-d3b4-4975-818a-22cee9d30dce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:53.203360 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.203184 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-home" pod "1a5f18bf-96ba-4edf-a758-206a1bd926a3" (UID: "1a5f18bf-96ba-4edf-a758-206a1bd926a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:53.204813 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.202744 4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-model-cache\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:53.205237 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.205121 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-home" pod "0679a60b-f854-49dd-a87d-3ea640cac995" (UID: "0679a60b-f854-49dd-a87d-3ea640cac995"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:53.206141 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.205974 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0679a60b-f854-49dd-a87d-3ea640cac995-kube-api-access-5pf9l" pod "0679a60b-f854-49dd-a87d-3ea640cac995" (UID: "0679a60b-f854-49dd-a87d-3ea640cac995"). InnerVolumeSpecName "kube-api-access-5pf9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:53.206141 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.206010 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a5f18bf-96ba-4edf-a758-206a1bd926a3-kube-api-access-65r8x" pod "1a5f18bf-96ba-4edf-a758-206a1bd926a3" (UID: "1a5f18bf-96ba-4edf-a758-206a1bd926a3"). InnerVolumeSpecName "kube-api-access-65r8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:53.206141 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.206096 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23b88e6-d3b4-4975-818a-22cee9d30dce-kube-api-access-t2rvw" pod "f23b88e6-d3b4-4975-818a-22cee9d30dce" (UID: "f23b88e6-d3b4-4975-818a-22cee9d30dce"). InnerVolumeSpecName "kube-api-access-t2rvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:53.206362 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.206269 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-dshm" pod "0679a60b-f854-49dd-a87d-3ea640cac995" (UID: "0679a60b-f854-49dd-a87d-3ea640cac995"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:53.206421 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.206382 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f23b88e6-d3b4-4975-818a-22cee9d30dce-tls-certs" pod "f23b88e6-d3b4-4975-818a-22cee9d30dce" (UID: "f23b88e6-d3b4-4975-818a-22cee9d30dce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:42:53.206421 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.206388 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-dshm" pod "1a5f18bf-96ba-4edf-a758-206a1bd926a3" (UID: "1a5f18bf-96ba-4edf-a758-206a1bd926a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:53.206515 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.206479 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a5f18bf-96ba-4edf-a758-206a1bd926a3-tls-certs" pod "1a5f18bf-96ba-4edf-a758-206a1bd926a3" (UID: "1a5f18bf-96ba-4edf-a758-206a1bd926a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:42:53.206921 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.206903 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-dshm" pod "f23b88e6-d3b4-4975-818a-22cee9d30dce" (UID: "f23b88e6-d3b4-4975-818a-22cee9d30dce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:53.207764 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.207745 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0679a60b-f854-49dd-a87d-3ea640cac995-tls-certs" pod "0679a60b-f854-49dd-a87d-3ea640cac995" (UID: "0679a60b-f854-49dd-a87d-3ea640cac995"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:42:53.305936 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.305907 4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-home\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:53.305936 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.305932 4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a5f18bf-96ba-4edf-a758-206a1bd926a3-tls-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:53.306167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.305943 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2rvw\" (UniqueName: \"kubernetes.io/projected/f23b88e6-d3b4-4975-818a-22cee9d30dce-kube-api-access-t2rvw\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:53.306167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.305953 4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-tmp-dir\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:53.306167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.305961 4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-dshm\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:53.306167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.305970 4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-dshm\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:53.306167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.305977 4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f23b88e6-d3b4-4975-818a-22cee9d30dce-tmp-dir\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:53.306167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.305985 4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a5f18bf-96ba-4edf-a758-206a1bd926a3-tmp-dir\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:53.306167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.305993 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65r8x\" (UniqueName: \"kubernetes.io/projected/1a5f18bf-96ba-4edf-a758-206a1bd926a3-kube-api-access-65r8x\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:53.306167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.306001 4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-dshm\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:53.306167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.306009 4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0679a60b-f854-49dd-a87d-3ea640cac995-tls-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:53.306167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.306016 4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0679a60b-f854-49dd-a87d-3ea640cac995-home\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:53.306167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.306023 4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f23b88e6-d3b4-4975-818a-22cee9d30dce-tls-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:53.306167 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.306030 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pf9l\" (UniqueName: \"kubernetes.io/projected/0679a60b-f854-49dd-a87d-3ea640cac995-kube-api-access-5pf9l\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:42:53.875490 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.875454 4531 generic.go:356] "Generic (PLEG): container finished" podID="1a5f18bf-96ba-4edf-a758-206a1bd926a3" containerID="9190ef4458cfdbc9586184bedbf49fe9c73e01228e2c297f4ba75f4b0d21fee4" exitCode=0 Oct 30 19:42:53.875981 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.875505 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" Oct 30 19:42:53.875981 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.875547 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" event={"ID":"1a5f18bf-96ba-4edf-a758-206a1bd926a3","Type":"ContainerDied","Data":"9190ef4458cfdbc9586184bedbf49fe9c73e01228e2c297f4ba75f4b0d21fee4"} Oct 30 19:42:53.875981 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.875589 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97" event={"ID":"1a5f18bf-96ba-4edf-a758-206a1bd926a3","Type":"ContainerDied","Data":"0818bca23fd7e435589fcfddf57f041692918f435791a954b24b36e37415b271"} Oct 30 19:42:53.875981 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.875611 4531 scope.go:122] "RemoveContainer" containerID="9190ef4458cfdbc9586184bedbf49fe9c73e01228e2c297f4ba75f4b0d21fee4" Oct 30 19:42:53.877006 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.876985 4531 generic.go:356] "Generic (PLEG): container finished" podID="f23b88e6-d3b4-4975-818a-22cee9d30dce" containerID="8350fe722af29da10c0821946defc1c6e67f386563499b68f05fa2386ec5aee8" exitCode=0 Oct 30 19:42:53.877088 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.877038 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" Oct 30 19:42:53.877088 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.877062 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" event={"ID":"f23b88e6-d3b4-4975-818a-22cee9d30dce","Type":"ContainerDied","Data":"8350fe722af29da10c0821946defc1c6e67f386563499b68f05fa2386ec5aee8"} Oct 30 19:42:53.877236 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.877104 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx" event={"ID":"f23b88e6-d3b4-4975-818a-22cee9d30dce","Type":"ContainerDied","Data":"c5df9b46440a384e8d165fb78ff22282b5b153cc01872cff0ced57490c554634"} Oct 30 19:42:53.878683 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.878659 4531 generic.go:356] "Generic (PLEG): container finished" podID="0679a60b-f854-49dd-a87d-3ea640cac995" containerID="cc78196bd0fb9c1faa36e9bbf14fe4cde15e8a49b6b1fd45406222a53368427a" exitCode=0 Oct 30 19:42:53.878854 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.878695 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" event={"ID":"0679a60b-f854-49dd-a87d-3ea640cac995","Type":"ContainerDied","Data":"cc78196bd0fb9c1faa36e9bbf14fe4cde15e8a49b6b1fd45406222a53368427a"} Oct 30 19:42:53.878973 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.878868 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" event={"ID":"0679a60b-f854-49dd-a87d-3ea640cac995","Type":"ContainerDied","Data":"df901ee41d32b1faa7c89f470b36bb46478769f74b24223a095ec322656d0520"} Oct 30 19:42:53.878973 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.878715 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8" Oct 30 19:42:53.888655 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.888623 4531 scope.go:122] "RemoveContainer" containerID="9190ef4458cfdbc9586184bedbf49fe9c73e01228e2c297f4ba75f4b0d21fee4" Oct 30 19:42:53.891704 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:42:53.890760 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9190ef4458cfdbc9586184bedbf49fe9c73e01228e2c297f4ba75f4b0d21fee4\": container with ID starting with 9190ef4458cfdbc9586184bedbf49fe9c73e01228e2c297f4ba75f4b0d21fee4 not found: ID does not exist" containerID="9190ef4458cfdbc9586184bedbf49fe9c73e01228e2c297f4ba75f4b0d21fee4" Oct 30 19:42:53.891704 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.890804 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9190ef4458cfdbc9586184bedbf49fe9c73e01228e2c297f4ba75f4b0d21fee4"} err="failed to get container status \"9190ef4458cfdbc9586184bedbf49fe9c73e01228e2c297f4ba75f4b0d21fee4\": rpc error: code = NotFound desc = could not find container \"9190ef4458cfdbc9586184bedbf49fe9c73e01228e2c297f4ba75f4b0d21fee4\": container with ID starting with 9190ef4458cfdbc9586184bedbf49fe9c73e01228e2c297f4ba75f4b0d21fee4 not found: ID does not exist" Oct 30 19:42:53.891704 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.890835 4531 scope.go:122] "RemoveContainer" containerID="8350fe722af29da10c0821946defc1c6e67f386563499b68f05fa2386ec5aee8" Oct 30 19:42:53.902524 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.902499 4531 scope.go:122] "RemoveContainer" containerID="8350fe722af29da10c0821946defc1c6e67f386563499b68f05fa2386ec5aee8" Oct 30 19:42:53.902821 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:42:53.902800 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8350fe722af29da10c0821946defc1c6e67f386563499b68f05fa2386ec5aee8\": container with ID starting with 8350fe722af29da10c0821946defc1c6e67f386563499b68f05fa2386ec5aee8 not found: ID does not exist" containerID="8350fe722af29da10c0821946defc1c6e67f386563499b68f05fa2386ec5aee8" Oct 30 19:42:53.902867 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.902831 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8350fe722af29da10c0821946defc1c6e67f386563499b68f05fa2386ec5aee8"} err="failed to get container status \"8350fe722af29da10c0821946defc1c6e67f386563499b68f05fa2386ec5aee8\": rpc error: code = NotFound desc = could not find container \"8350fe722af29da10c0821946defc1c6e67f386563499b68f05fa2386ec5aee8\": container with ID starting with 8350fe722af29da10c0821946defc1c6e67f386563499b68f05fa2386ec5aee8 not found: ID does not exist" Oct 30 19:42:53.902867 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.902850 4531 scope.go:122] "RemoveContainer" containerID="cc78196bd0fb9c1faa36e9bbf14fe4cde15e8a49b6b1fd45406222a53368427a" Oct 30 19:42:53.912550 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.912527 4531 scope.go:122] "RemoveContainer" containerID="cc78196bd0fb9c1faa36e9bbf14fe4cde15e8a49b6b1fd45406222a53368427a" Oct 30 19:42:53.912883 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:42:53.912861 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc78196bd0fb9c1faa36e9bbf14fe4cde15e8a49b6b1fd45406222a53368427a\": container with ID starting with cc78196bd0fb9c1faa36e9bbf14fe4cde15e8a49b6b1fd45406222a53368427a not found: ID does not exist" containerID="cc78196bd0fb9c1faa36e9bbf14fe4cde15e8a49b6b1fd45406222a53368427a" Oct 30 19:42:53.912936 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:53.912895 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc78196bd0fb9c1faa36e9bbf14fe4cde15e8a49b6b1fd45406222a53368427a"} err="failed to get container status \"cc78196bd0fb9c1faa36e9bbf14fe4cde15e8a49b6b1fd45406222a53368427a\": rpc error: code = NotFound desc = could not find container \"cc78196bd0fb9c1faa36e9bbf14fe4cde15e8a49b6b1fd45406222a53368427a\": container with ID starting with cc78196bd0fb9c1faa36e9bbf14fe4cde15e8a49b6b1fd45406222a53368427a not found: ID does not exist" Oct 30 19:42:54.040177 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:54.040084 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx"] Oct 30 19:42:54.046134 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:54.046094 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-fd61e3/distinct-w1-fd61e3-kserve-5c86f99b4c-jwfkx"] Oct 30 19:42:54.097458 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:54.097424 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f23b88e6-d3b4-4975-818a-22cee9d30dce" path="/var/lib/kubelet/pods/f23b88e6-d3b4-4975-818a-22cee9d30dce/volumes" Oct 30 19:42:54.155985 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:54.155896 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97"] Oct 30 19:42:54.162264 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:54.162236 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-fd61e3/simulator-w1-fd61e3-kserve-649c45966f-hrl97"] Oct 30 19:42:54.271796 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:54.271761 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8"] Oct 30 19:42:54.278306 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:54.278268 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-fd61e3/premium-w1-fd61e3-kserve-584cf676-dbzx8"] Oct 30 19:42:56.097375 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:56.097335 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0679a60b-f854-49dd-a87d-3ea640cac995" path="/var/lib/kubelet/pods/0679a60b-f854-49dd-a87d-3ea640cac995/volumes" Oct 30 19:42:56.097782 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:42:56.097767 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a5f18bf-96ba-4edf-a758-206a1bd926a3" path="/var/lib/kubelet/pods/1a5f18bf-96ba-4edf-a758-206a1bd926a3/volumes" Oct 30 19:42:56.397300 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:42:56.397221 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:43:01.137524 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.137479 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7529a119-chzjh"] Oct 30 19:43:01.138044 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.138014 4531 cpu_manager.go:414] "containerMap: removing container" podUID="0679a60b-f854-49dd-a87d-3ea640cac995" containerName="main" Oct 30 19:43:01.138044 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.138033 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0679a60b-f854-49dd-a87d-3ea640cac995" containerName="main" Oct 30 19:43:01.138201 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.138060 4531 cpu_manager.go:414] "containerMap: removing container" podUID="f23b88e6-d3b4-4975-818a-22cee9d30dce" containerName="main" Oct 30 19:43:01.138201 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.138067 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f23b88e6-d3b4-4975-818a-22cee9d30dce" containerName="main" Oct 30 19:43:01.138201 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.138080 4531 cpu_manager.go:414] "containerMap: removing container" podUID="0ea84ad8-53fd-4c3d-a32f-d202b3075ac1" containerName="istio-proxy" Oct 30 19:43:01.138201 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.138088 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ea84ad8-53fd-4c3d-a32f-d202b3075ac1" containerName="istio-proxy" Oct 30 19:43:01.138201 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.138100 4531 cpu_manager.go:414] "containerMap: removing container" podUID="1a5f18bf-96ba-4edf-a758-206a1bd926a3" containerName="main" Oct 30 19:43:01.138201 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.138107 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a5f18bf-96ba-4edf-a758-206a1bd926a3" containerName="main" Oct 30 19:43:01.138201 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.138156 4531 cpu_manager.go:414] "containerMap: removing container" podUID="b3ad1db0-bade-4eb9-9033-abdee1536bde" containerName="revoke-keys" Oct 30 19:43:01.138201 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.138166 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3ad1db0-bade-4eb9-9033-abdee1536bde" containerName="revoke-keys" Oct 30 19:43:01.138610 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.138272 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0679a60b-f854-49dd-a87d-3ea640cac995" containerName="main" Oct 30 19:43:01.138610 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.138289 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f23b88e6-d3b4-4975-818a-22cee9d30dce" containerName="main" Oct 30 19:43:01.138610 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.138301 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a5f18bf-96ba-4edf-a758-206a1bd926a3" containerName="main" Oct 30 19:43:01.138610 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.138312 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3ad1db0-bade-4eb9-9033-abdee1536bde" containerName="revoke-keys" Oct 30 19:43:01.143240 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.143217 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7529a119-chzjh" Oct 30 19:43:01.180389 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.180350 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bfb7d5b-9b07-48a1-8982-c6d81e47618c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7529a119-chzjh\" (UID: \"5bfb7d5b-9b07-48a1-8982-c6d81e47618c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7529a119-chzjh" Oct 30 19:43:01.180617 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.180407 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bfb7d5b-9b07-48a1-8982-c6d81e47618c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7529a119-chzjh\" (UID: \"5bfb7d5b-9b07-48a1-8982-c6d81e47618c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7529a119-chzjh" Oct 30 19:43:01.180617 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.180525 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjfpz\" (UniqueName: \"kubernetes.io/projected/5bfb7d5b-9b07-48a1-8982-c6d81e47618c-kube-api-access-gjfpz\") pod \"maas-api-revoke-sub-7529a119-chzjh\" (UID: \"5bfb7d5b-9b07-48a1-8982-c6d81e47618c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7529a119-chzjh" Oct 30 19:43:01.200381 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.200346 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7529a119-chzjh"] Oct 30 19:43:01.247217 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.247167 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:43:01.247217 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.247178 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:43:01.247487 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.247177 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:43:01.247487 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.247415 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:43:01.249345 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.249321 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81d557ab-q46p4"] Oct 30 19:43:01.252810 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.252790 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d557ab-q46p4" Oct 30 19:43:01.281177 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.281123 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjfpz\" (UniqueName: \"kubernetes.io/projected/5bfb7d5b-9b07-48a1-8982-c6d81e47618c-kube-api-access-gjfpz\") pod \"maas-api-revoke-sub-7529a119-chzjh\" (UID: \"5bfb7d5b-9b07-48a1-8982-c6d81e47618c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7529a119-chzjh" Oct 30 19:43:01.281428 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.281406 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b173689-c1d4-419e-bb7e-2ac0afd4f455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81d557ab-q46p4\" (UID: \"4b173689-c1d4-419e-bb7e-2ac0afd4f455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d557ab-q46p4" Oct 30 19:43:01.281575 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.281439 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2czff\" (UniqueName: \"kubernetes.io/projected/4b173689-c1d4-419e-bb7e-2ac0afd4f455-kube-api-access-2czff\") pod \"maas-api-revoke-sub-81d557ab-q46p4\" (UID: \"4b173689-c1d4-419e-bb7e-2ac0afd4f455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d557ab-q46p4" Oct 30 19:43:01.281637 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.281567 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bfb7d5b-9b07-48a1-8982-c6d81e47618c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7529a119-chzjh\" (UID: \"5bfb7d5b-9b07-48a1-8982-c6d81e47618c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7529a119-chzjh" Oct 30 19:43:01.281637 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.281615 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b173689-c1d4-419e-bb7e-2ac0afd4f455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81d557ab-q46p4\" (UID: \"4b173689-c1d4-419e-bb7e-2ac0afd4f455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d557ab-q46p4" Oct 30 19:43:01.281729 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.281668 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bfb7d5b-9b07-48a1-8982-c6d81e47618c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7529a119-chzjh\" (UID: \"5bfb7d5b-9b07-48a1-8982-c6d81e47618c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7529a119-chzjh" Oct 30 19:43:01.282247 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.282214 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bfb7d5b-9b07-48a1-8982-c6d81e47618c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7529a119-chzjh\" (UID: \"5bfb7d5b-9b07-48a1-8982-c6d81e47618c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7529a119-chzjh" Oct 30 19:43:01.284419 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.284386 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bfb7d5b-9b07-48a1-8982-c6d81e47618c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7529a119-chzjh\" (UID: \"5bfb7d5b-9b07-48a1-8982-c6d81e47618c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7529a119-chzjh" Oct 30 19:43:01.319680 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.319631 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81d557ab-q46p4"] Oct 30 19:43:01.342542 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.342505 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjfpz\" (UniqueName: \"kubernetes.io/projected/5bfb7d5b-9b07-48a1-8982-c6d81e47618c-kube-api-access-gjfpz\") pod \"maas-api-revoke-sub-7529a119-chzjh\" (UID: \"5bfb7d5b-9b07-48a1-8982-c6d81e47618c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7529a119-chzjh" Oct 30 19:43:01.383054 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.383002 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b173689-c1d4-419e-bb7e-2ac0afd4f455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81d557ab-q46p4\" (UID: \"4b173689-c1d4-419e-bb7e-2ac0afd4f455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d557ab-q46p4" Oct 30 19:43:01.383306 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.383062 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2czff\" (UniqueName: \"kubernetes.io/projected/4b173689-c1d4-419e-bb7e-2ac0afd4f455-kube-api-access-2czff\") pod \"maas-api-revoke-sub-81d557ab-q46p4\" (UID: \"4b173689-c1d4-419e-bb7e-2ac0afd4f455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d557ab-q46p4" Oct 30 19:43:01.383306 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.383167 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b173689-c1d4-419e-bb7e-2ac0afd4f455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81d557ab-q46p4\" (UID: \"4b173689-c1d4-419e-bb7e-2ac0afd4f455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d557ab-q46p4" Oct 30 19:43:01.383796 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.383771 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b173689-c1d4-419e-bb7e-2ac0afd4f455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81d557ab-q46p4\" (UID: \"4b173689-c1d4-419e-bb7e-2ac0afd4f455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d557ab-q46p4" Oct 30 19:43:01.385900 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.385851 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b173689-c1d4-419e-bb7e-2ac0afd4f455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81d557ab-q46p4\" (UID: \"4b173689-c1d4-419e-bb7e-2ac0afd4f455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d557ab-q46p4" Oct 30 19:43:01.448937 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.448860 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2czff\" (UniqueName: \"kubernetes.io/projected/4b173689-c1d4-419e-bb7e-2ac0afd4f455-kube-api-access-2czff\") pod \"maas-api-revoke-sub-81d557ab-q46p4\" (UID: \"4b173689-c1d4-419e-bb7e-2ac0afd4f455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d557ab-q46p4" Oct 30 19:43:01.455482 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.455461 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7529a119-chzjh" Oct 30 19:43:01.567556 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.567513 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d557ab-q46p4" Oct 30 19:43:01.846983 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.846956 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7529a119-chzjh"] Oct 30 19:43:01.849345 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:43:01.849308 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bfb7d5b_9b07_48a1_8982_c6d81e47618c.slice/crio-33fa8681345c00283596a4a85e67333059f9e0df23694382640362339812302c WatchSource:0}: Error finding container 33fa8681345c00283596a4a85e67333059f9e0df23694382640362339812302c: Status 404 returned error can't find the container with id 33fa8681345c00283596a4a85e67333059f9e0df23694382640362339812302c Oct 30 19:43:01.928202 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.928167 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7529a119-chzjh" event={"ID":"5bfb7d5b-9b07-48a1-8982-c6d81e47618c","Type":"ContainerStarted","Data":"33fa8681345c00283596a4a85e67333059f9e0df23694382640362339812302c"} Oct 30 19:43:01.958710 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:01.958685 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81d557ab-q46p4"] Oct 30 19:43:01.960751 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:43:01.960726 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b173689_c1d4_419e_bb7e_2ac0afd4f455.slice/crio-3dab700fdbec74fd8ddc8ea694eb048ed54d9dee613ed8342d303166e6a5420c WatchSource:0}: Error finding container 3dab700fdbec74fd8ddc8ea694eb048ed54d9dee613ed8342d303166e6a5420c: Status 404 returned error can't find the container with id 3dab700fdbec74fd8ddc8ea694eb048ed54d9dee613ed8342d303166e6a5420c Oct 30 19:43:02.674613 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:02.674586 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53debdd7-wkdwc"] Oct 30 19:43:02.678163 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:02.678120 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53debdd7-wkdwc" Oct 30 19:43:02.697717 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:02.697679 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc8mx\" (UniqueName: \"kubernetes.io/projected/1b7c30b8-fc0a-43be-81d7-23b089c6881e-kube-api-access-kc8mx\") pod \"maas-api-revoke-sub-53debdd7-wkdwc\" (UID: \"1b7c30b8-fc0a-43be-81d7-23b089c6881e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53debdd7-wkdwc" Oct 30 19:43:02.697913 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:02.697742 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7c30b8-fc0a-43be-81d7-23b089c6881e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53debdd7-wkdwc\" (UID: \"1b7c30b8-fc0a-43be-81d7-23b089c6881e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53debdd7-wkdwc" Oct 30 19:43:02.697989 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:02.697905 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7c30b8-fc0a-43be-81d7-23b089c6881e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53debdd7-wkdwc\" (UID: \"1b7c30b8-fc0a-43be-81d7-23b089c6881e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53debdd7-wkdwc" Oct 30 19:43:02.738170 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:02.738123 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53debdd7-wkdwc"] Oct 30 19:43:02.802325 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:02.799082 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc8mx\" (UniqueName: \"kubernetes.io/projected/1b7c30b8-fc0a-43be-81d7-23b089c6881e-kube-api-access-kc8mx\") pod \"maas-api-revoke-sub-53debdd7-wkdwc\" (UID: \"1b7c30b8-fc0a-43be-81d7-23b089c6881e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53debdd7-wkdwc" Oct 30 19:43:02.802325 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:02.799184 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7c30b8-fc0a-43be-81d7-23b089c6881e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53debdd7-wkdwc\" (UID: \"1b7c30b8-fc0a-43be-81d7-23b089c6881e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53debdd7-wkdwc" Oct 30 19:43:02.802325 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:02.799289 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7c30b8-fc0a-43be-81d7-23b089c6881e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53debdd7-wkdwc\" (UID: \"1b7c30b8-fc0a-43be-81d7-23b089c6881e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53debdd7-wkdwc" Oct 30 19:43:02.802325 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:02.800804 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7c30b8-fc0a-43be-81d7-23b089c6881e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53debdd7-wkdwc\" (UID: \"1b7c30b8-fc0a-43be-81d7-23b089c6881e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53debdd7-wkdwc" Oct 30 19:43:02.802695 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:02.802565 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7c30b8-fc0a-43be-81d7-23b089c6881e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53debdd7-wkdwc\" (UID: \"1b7c30b8-fc0a-43be-81d7-23b089c6881e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53debdd7-wkdwc" Oct 30 19:43:02.862969 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:02.862938 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc8mx\" (UniqueName: \"kubernetes.io/projected/1b7c30b8-fc0a-43be-81d7-23b089c6881e-kube-api-access-kc8mx\") pod \"maas-api-revoke-sub-53debdd7-wkdwc\" (UID: \"1b7c30b8-fc0a-43be-81d7-23b089c6881e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53debdd7-wkdwc" Oct 30 19:43:02.935669 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:02.935564 4531 generic.go:356] "Generic (PLEG): container finished" podID="4b173689-c1d4-419e-bb7e-2ac0afd4f455" containerID="0df188977a90e1ad8f45d770536db88e08a7b98cf1989c8323236ad2cad7f9b4" exitCode=0 Oct 30 19:43:02.935846 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:02.935661 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d557ab-q46p4" event={"ID":"4b173689-c1d4-419e-bb7e-2ac0afd4f455","Type":"ContainerDied","Data":"0df188977a90e1ad8f45d770536db88e08a7b98cf1989c8323236ad2cad7f9b4"} Oct 30 19:43:02.935846 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:02.935711 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d557ab-q46p4" event={"ID":"4b173689-c1d4-419e-bb7e-2ac0afd4f455","Type":"ContainerStarted","Data":"3dab700fdbec74fd8ddc8ea694eb048ed54d9dee613ed8342d303166e6a5420c"} Oct 30 19:43:02.937302 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:02.937279 4531 generic.go:356] "Generic (PLEG): container finished" podID="5bfb7d5b-9b07-48a1-8982-c6d81e47618c" containerID="dbb82233e51bc6bc68e951378dc4902058531eee10c4260a830b5319e810a859" exitCode=0 Oct 30 19:43:02.937424 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:02.937332 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7529a119-chzjh" event={"ID":"5bfb7d5b-9b07-48a1-8982-c6d81e47618c","Type":"ContainerDied","Data":"dbb82233e51bc6bc68e951378dc4902058531eee10c4260a830b5319e810a859"} Oct 30 19:43:03.059403 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:03.059365 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53debdd7-wkdwc" Oct 30 19:43:03.310292 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:03.310258 4531 scope.go:122] "RemoveContainer" containerID="14893e9b0337d55992ed002317bf01bcb5be32fd05ada85851645bd0c203b865" Oct 30 19:43:03.321558 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:03.321530 4531 scope.go:122] "RemoveContainer" containerID="fa645bc3bdfd2896836bb42e73ed845cbe6a236a785599e7425c02a2df3d0187" Oct 30 19:43:03.436784 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:03.436759 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53debdd7-wkdwc"] Oct 30 19:43:03.439212 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:43:03.439182 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7c30b8_fc0a_43be_81d7_23b089c6881e.slice/crio-5438f76df52083764881c325c4080d5f0d79d0c66d334bf385df7167f7c57f9f WatchSource:0}: Error finding container 5438f76df52083764881c325c4080d5f0d79d0c66d334bf385df7167f7c57f9f: Status 404 returned error can't find the container with id 5438f76df52083764881c325c4080d5f0d79d0c66d334bf385df7167f7c57f9f Oct 30 19:43:03.650602 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:03.650566 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4"] Oct 30 19:43:03.655427 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:03.655389 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-4a7a52-8j5g4"] Oct 30 19:43:03.944464 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:03.944375 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53debdd7-wkdwc" event={"ID":"1b7c30b8-fc0a-43be-81d7-23b089c6881e","Type":"ContainerStarted","Data":"587a374c5ef15a06bb99a68041eee744a02b1ec33190d0c164fa9e7f9260b783"} Oct 30 19:43:03.944464 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:03.944432 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53debdd7-wkdwc" event={"ID":"1b7c30b8-fc0a-43be-81d7-23b089c6881e","Type":"ContainerStarted","Data":"5438f76df52083764881c325c4080d5f0d79d0c66d334bf385df7167f7c57f9f"} Oct 30 19:43:04.066356 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.066288 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53debdd7-wkdwc" podStartSLOduration=2.066266643 podStartE2EDuration="2.066266643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:04.062926012 +0000 UTC m=+4144.090030971" watchObservedRunningTime="2026-10-30 19:43:04.066266643 +0000 UTC m=+4144.093371598" Oct 30 19:43:04.098423 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.098394 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="498eff0d-6631-4255-a0f4-8491a9174258" path="/var/lib/kubelet/pods/498eff0d-6631-4255-a0f4-8491a9174258/volumes" Oct 30 19:43:04.101517 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.101491 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7529a119-chzjh" Oct 30 19:43:04.119065 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.119034 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d557ab-q46p4" Oct 30 19:43:04.212120 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.212031 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b173689-c1d4-419e-bb7e-2ac0afd4f455-kube-api-access-2czff\" (UniqueName: \"kubernetes.io/projected/4b173689-c1d4-419e-bb7e-2ac0afd4f455-kube-api-access-2czff\") pod \"4b173689-c1d4-419e-bb7e-2ac0afd4f455\" (UID: \"4b173689-c1d4-419e-bb7e-2ac0afd4f455\") " Oct 30 19:43:04.212334 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.212119 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b173689-c1d4-419e-bb7e-2ac0afd4f455-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b173689-c1d4-419e-bb7e-2ac0afd4f455-maas-api-openshift-ca\") pod \"4b173689-c1d4-419e-bb7e-2ac0afd4f455\" (UID: \"4b173689-c1d4-419e-bb7e-2ac0afd4f455\") " Oct 30 19:43:04.212334 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.212208 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bfb7d5b-9b07-48a1-8982-c6d81e47618c-kube-api-access-gjfpz\" (UniqueName: \"kubernetes.io/projected/5bfb7d5b-9b07-48a1-8982-c6d81e47618c-kube-api-access-gjfpz\") pod \"5bfb7d5b-9b07-48a1-8982-c6d81e47618c\" (UID: \"5bfb7d5b-9b07-48a1-8982-c6d81e47618c\") " Oct 30 19:43:04.212334 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.212242 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b173689-c1d4-419e-bb7e-2ac0afd4f455-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b173689-c1d4-419e-bb7e-2ac0afd4f455-maas-api-xks-ca\") pod \"4b173689-c1d4-419e-bb7e-2ac0afd4f455\" (UID: \"4b173689-c1d4-419e-bb7e-2ac0afd4f455\") " Oct 30 19:43:04.212334 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.212284 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bfb7d5b-9b07-48a1-8982-c6d81e47618c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bfb7d5b-9b07-48a1-8982-c6d81e47618c-maas-api-xks-ca\") pod \"5bfb7d5b-9b07-48a1-8982-c6d81e47618c\" (UID: \"5bfb7d5b-9b07-48a1-8982-c6d81e47618c\") " Oct 30 19:43:04.212334 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.212318 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bfb7d5b-9b07-48a1-8982-c6d81e47618c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bfb7d5b-9b07-48a1-8982-c6d81e47618c-maas-api-openshift-ca\") pod \"5bfb7d5b-9b07-48a1-8982-c6d81e47618c\" (UID: \"5bfb7d5b-9b07-48a1-8982-c6d81e47618c\") " Oct 30 19:43:04.213121 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.212445 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b173689-c1d4-419e-bb7e-2ac0afd4f455-maas-api-openshift-ca" pod "4b173689-c1d4-419e-bb7e-2ac0afd4f455" (UID: "4b173689-c1d4-419e-bb7e-2ac0afd4f455"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:04.213121 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.212691 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b173689-c1d4-419e-bb7e-2ac0afd4f455-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:04.213121 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.212978 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bfb7d5b-9b07-48a1-8982-c6d81e47618c-maas-api-openshift-ca" pod "5bfb7d5b-9b07-48a1-8982-c6d81e47618c" (UID: "5bfb7d5b-9b07-48a1-8982-c6d81e47618c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:04.214687 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.214659 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b173689-c1d4-419e-bb7e-2ac0afd4f455-maas-api-xks-ca" pod "4b173689-c1d4-419e-bb7e-2ac0afd4f455" (UID: "4b173689-c1d4-419e-bb7e-2ac0afd4f455"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:43:04.214793 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.214687 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfb7d5b-9b07-48a1-8982-c6d81e47618c-kube-api-access-gjfpz" pod "5bfb7d5b-9b07-48a1-8982-c6d81e47618c" (UID: "5bfb7d5b-9b07-48a1-8982-c6d81e47618c"). InnerVolumeSpecName "kube-api-access-gjfpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:04.215015 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.214996 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b173689-c1d4-419e-bb7e-2ac0afd4f455-kube-api-access-2czff" pod "4b173689-c1d4-419e-bb7e-2ac0afd4f455" (UID: "4b173689-c1d4-419e-bb7e-2ac0afd4f455"). InnerVolumeSpecName "kube-api-access-2czff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:04.215085 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.215008 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bfb7d5b-9b07-48a1-8982-c6d81e47618c-maas-api-xks-ca" pod "5bfb7d5b-9b07-48a1-8982-c6d81e47618c" (UID: "5bfb7d5b-9b07-48a1-8982-c6d81e47618c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:43:04.314119 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.314075 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b173689-c1d4-419e-bb7e-2ac0afd4f455-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:04.314119 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.314114 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bfb7d5b-9b07-48a1-8982-c6d81e47618c-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:04.314381 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.314163 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bfb7d5b-9b07-48a1-8982-c6d81e47618c-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:04.314381 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.314178 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2czff\" (UniqueName: \"kubernetes.io/projected/4b173689-c1d4-419e-bb7e-2ac0afd4f455-kube-api-access-2czff\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:04.314381 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.314191 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjfpz\" (UniqueName: \"kubernetes.io/projected/5bfb7d5b-9b07-48a1-8982-c6d81e47618c-kube-api-access-gjfpz\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:04.814224 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.814172 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" podUID="9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e" containerName="main" containerID="cri-o://db5277d35bda9ea46c09e1c3dadcca354b37eff8d59a82ace86ec474d2e42c66" gracePeriod=45 Oct 30 19:43:04.867982 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.867935 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" podUID="471f1171-e71f-4da2-862d-5d232c45d6cc" containerName="main" containerID="cri-o://0eacf94a6e4785a277442cd14bbb868e2264ddd5e4916191dfa5fff8b523c053" gracePeriod=45 Oct 30 19:43:04.869945 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.869907 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" podUID="586f370d-9916-43a9-ac7d-e77cc3fcb92f" containerName="main" containerID="cri-o://3cbc4c5ffbf8572bf9fdff80aa73ebaca28dc4ea0934d2aebe78331cf02556dc" gracePeriod=45 Oct 30 19:43:04.873482 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.873434 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" podUID="5a868137-eb41-433c-ade6-58eb8c96d623" containerName="main" containerID="cri-o://ffaf31cc4895a3a2b0e24725687a341fa9c99a439b7600d812b62278b55b5e9e" gracePeriod=45 Oct 30 19:43:04.952100 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.952071 4531 generic.go:356] "Generic (PLEG): container finished" podID="9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e" containerID="db5277d35bda9ea46c09e1c3dadcca354b37eff8d59a82ace86ec474d2e42c66" exitCode=0 Oct 30 19:43:04.952478 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.952153 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" event={"ID":"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e","Type":"ContainerDied","Data":"db5277d35bda9ea46c09e1c3dadcca354b37eff8d59a82ace86ec474d2e42c66"} Oct 30 19:43:04.954154 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.954114 4531 generic.go:356] "Generic (PLEG): container finished" podID="1b7c30b8-fc0a-43be-81d7-23b089c6881e" containerID="587a374c5ef15a06bb99a68041eee744a02b1ec33190d0c164fa9e7f9260b783" exitCode=0 Oct 30 19:43:04.954245 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.954162 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53debdd7-wkdwc" event={"ID":"1b7c30b8-fc0a-43be-81d7-23b089c6881e","Type":"ContainerDied","Data":"587a374c5ef15a06bb99a68041eee744a02b1ec33190d0c164fa9e7f9260b783"} Oct 30 19:43:04.955457 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.955437 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d557ab-q46p4" Oct 30 19:43:04.955561 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.955470 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d557ab-q46p4" event={"ID":"4b173689-c1d4-419e-bb7e-2ac0afd4f455","Type":"ContainerDied","Data":"3dab700fdbec74fd8ddc8ea694eb048ed54d9dee613ed8342d303166e6a5420c"} Oct 30 19:43:04.955561 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.955515 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dab700fdbec74fd8ddc8ea694eb048ed54d9dee613ed8342d303166e6a5420c" Oct 30 19:43:04.956669 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.956651 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7529a119-chzjh" event={"ID":"5bfb7d5b-9b07-48a1-8982-c6d81e47618c","Type":"ContainerDied","Data":"33fa8681345c00283596a4a85e67333059f9e0df23694382640362339812302c"} Oct 30 19:43:04.956669 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.956664 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7529a119-chzjh" Oct 30 19:43:04.956794 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:04.956688 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33fa8681345c00283596a4a85e67333059f9e0df23694382640362339812302c" Oct 30 19:43:05.154731 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.154700 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 19:43:05.174213 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.174157 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" Oct 30 19:43:05.195595 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.195568 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" Oct 30 19:43:05.202730 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.202701 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 19:43:05.223833 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.223801 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-home\" (UniqueName: \"kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-home\") pod \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\" (UID: \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\") " Oct 30 19:43:05.224593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.223846 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-tls-certs\" (UniqueName: \"kubernetes.io/secret/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-tls-certs\") pod \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\" (UID: \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\") " Oct 30 19:43:05.224593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.223890 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-dshm\") pod \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\" (UID: \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\") " Oct 30 19:43:05.224593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.223922 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-dshm\") pod \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\" (UID: \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\") " Oct 30 19:43:05.224593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.223947 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-kube-api-access-p7zsj\" (UniqueName: \"kubernetes.io/projected/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-kube-api-access-p7zsj\") pod \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\" (UID: \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\") " Oct 30 19:43:05.224593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.224013 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-home\" (UniqueName: \"kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-home\") pod \"471f1171-e71f-4da2-862d-5d232c45d6cc\" (UID: \"471f1171-e71f-4da2-862d-5d232c45d6cc\") " Oct 30 19:43:05.224593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.224050 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/586f370d-9916-43a9-ac7d-e77cc3fcb92f-kube-api-access-nf94x\" (UniqueName: \"kubernetes.io/projected/586f370d-9916-43a9-ac7d-e77cc3fcb92f-kube-api-access-nf94x\") pod \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\" (UID: \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\") " Oct 30 19:43:05.224593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.224081 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-model-cache\") pod \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\" (UID: \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\") " Oct 30 19:43:05.224593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.224103 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-home" pod "586f370d-9916-43a9-ac7d-e77cc3fcb92f" (UID: "586f370d-9916-43a9-ac7d-e77cc3fcb92f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:05.224593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.224121 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/586f370d-9916-43a9-ac7d-e77cc3fcb92f-tls-certs\" (UniqueName: \"kubernetes.io/secret/586f370d-9916-43a9-ac7d-e77cc3fcb92f-tls-certs\") pod \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\" (UID: \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\") " Oct 30 19:43:05.224593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.224180 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-tmp-dir\") pod \"471f1171-e71f-4da2-862d-5d232c45d6cc\" (UID: \"471f1171-e71f-4da2-862d-5d232c45d6cc\") " Oct 30 19:43:05.224593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.224226 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-home\" (UniqueName: \"kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-home\") pod \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\" (UID: \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\") " Oct 30 19:43:05.224593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.224267 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-tmp-dir\") pod \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\" (UID: \"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e\") " Oct 30 19:43:05.224593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.224292 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/471f1171-e71f-4da2-862d-5d232c45d6cc-tls-certs\" (UniqueName: \"kubernetes.io/secret/471f1171-e71f-4da2-862d-5d232c45d6cc-tls-certs\") pod \"471f1171-e71f-4da2-862d-5d232c45d6cc\" (UID: \"471f1171-e71f-4da2-862d-5d232c45d6cc\") " Oct 30 19:43:05.224593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.224332 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-model-cache\") pod \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\" (UID: \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\") " Oct 30 19:43:05.224593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.224374 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-model-cache\") pod \"471f1171-e71f-4da2-862d-5d232c45d6cc\" (UID: \"471f1171-e71f-4da2-862d-5d232c45d6cc\") " Oct 30 19:43:05.224593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.224409 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/471f1171-e71f-4da2-862d-5d232c45d6cc-kube-api-access-25mxq\" (UniqueName: \"kubernetes.io/projected/471f1171-e71f-4da2-862d-5d232c45d6cc-kube-api-access-25mxq\") pod \"471f1171-e71f-4da2-862d-5d232c45d6cc\" (UID: \"471f1171-e71f-4da2-862d-5d232c45d6cc\") " Oct 30 19:43:05.224593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.224450 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-tmp-dir\") pod \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\" (UID: \"586f370d-9916-43a9-ac7d-e77cc3fcb92f\") " Oct 30 19:43:05.224593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.224480 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-dshm\") pod \"471f1171-e71f-4da2-862d-5d232c45d6cc\" (UID: \"471f1171-e71f-4da2-862d-5d232c45d6cc\") " Oct 30 19:43:05.224593 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.224521 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-tmp-dir" pod "471f1171-e71f-4da2-862d-5d232c45d6cc" (UID: "471f1171-e71f-4da2-862d-5d232c45d6cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:05.227854 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.224693 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-model-cache" pod "586f370d-9916-43a9-ac7d-e77cc3fcb92f" (UID: "586f370d-9916-43a9-ac7d-e77cc3fcb92f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:05.227854 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.224870 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-home" pod "471f1171-e71f-4da2-862d-5d232c45d6cc" (UID: "471f1171-e71f-4da2-862d-5d232c45d6cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:05.227854 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.224873 4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-tmp-dir\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.227854 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.224912 4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-model-cache\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.227854 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.224944 4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-home\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.227854 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.225064 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-model-cache" pod "9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e" (UID: "9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:05.227854 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.225300 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-model-cache" pod "471f1171-e71f-4da2-862d-5d232c45d6cc" (UID: "471f1171-e71f-4da2-862d-5d232c45d6cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:05.227854 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.225355 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-home" pod "9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e" (UID: "9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:05.227854 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.225717 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-tmp-dir" pod "586f370d-9916-43a9-ac7d-e77cc3fcb92f" (UID: "586f370d-9916-43a9-ac7d-e77cc3fcb92f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:05.227854 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.225742 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-tmp-dir" pod "9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e" (UID: "9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:05.228454 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.228303 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-kube-api-access-p7zsj" pod "9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e" (UID: "9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e"). InnerVolumeSpecName "kube-api-access-p7zsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:05.228544 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.228519 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-dshm" pod "586f370d-9916-43a9-ac7d-e77cc3fcb92f" (UID: "586f370d-9916-43a9-ac7d-e77cc3fcb92f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:05.228610 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.228571 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/586f370d-9916-43a9-ac7d-e77cc3fcb92f-kube-api-access-nf94x" pod "586f370d-9916-43a9-ac7d-e77cc3fcb92f" (UID: "586f370d-9916-43a9-ac7d-e77cc3fcb92f"). InnerVolumeSpecName "kube-api-access-nf94x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:05.228664 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.228604 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/471f1171-e71f-4da2-862d-5d232c45d6cc-tls-certs" pod "471f1171-e71f-4da2-862d-5d232c45d6cc" (UID: "471f1171-e71f-4da2-862d-5d232c45d6cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:43:05.228664 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.228633 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-tls-certs" pod "9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e" (UID: "9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:43:05.228920 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.228872 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-dshm" pod "9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e" (UID: "9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:05.229173 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.229140 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-dshm" pod "471f1171-e71f-4da2-862d-5d232c45d6cc" (UID: "471f1171-e71f-4da2-862d-5d232c45d6cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:05.229510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.229490 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/471f1171-e71f-4da2-862d-5d232c45d6cc-kube-api-access-25mxq" pod "471f1171-e71f-4da2-862d-5d232c45d6cc" (UID: "471f1171-e71f-4da2-862d-5d232c45d6cc"). InnerVolumeSpecName "kube-api-access-25mxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:05.229745 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.229727 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/586f370d-9916-43a9-ac7d-e77cc3fcb92f-tls-certs" pod "586f370d-9916-43a9-ac7d-e77cc3fcb92f" (UID: "586f370d-9916-43a9-ac7d-e77cc3fcb92f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:43:05.326266 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.326231 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a868137-eb41-433c-ade6-58eb8c96d623-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a868137-eb41-433c-ade6-58eb8c96d623-tls-certs\") pod \"5a868137-eb41-433c-ade6-58eb8c96d623\" (UID: \"5a868137-eb41-433c-ade6-58eb8c96d623\") " Oct 30 19:43:05.326470 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.326330 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-home\" (UniqueName: \"kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-home\") pod \"5a868137-eb41-433c-ade6-58eb8c96d623\" (UID: \"5a868137-eb41-433c-ade6-58eb8c96d623\") " Oct 30 19:43:05.326470 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.326368 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a868137-eb41-433c-ade6-58eb8c96d623-kube-api-access-k94hr\" (UniqueName: \"kubernetes.io/projected/5a868137-eb41-433c-ade6-58eb8c96d623-kube-api-access-k94hr\") pod \"5a868137-eb41-433c-ade6-58eb8c96d623\" (UID: \"5a868137-eb41-433c-ade6-58eb8c96d623\") " Oct 30 19:43:05.326470 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.326400 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-tmp-dir\") pod \"5a868137-eb41-433c-ade6-58eb8c96d623\" (UID: \"5a868137-eb41-433c-ade6-58eb8c96d623\") " Oct 30 19:43:05.326470 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.326450 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-dshm\") pod \"5a868137-eb41-433c-ade6-58eb8c96d623\" (UID: \"5a868137-eb41-433c-ade6-58eb8c96d623\") " Oct 30 19:43:05.326690 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.326529 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-model-cache\") pod \"5a868137-eb41-433c-ade6-58eb8c96d623\" (UID: \"5a868137-eb41-433c-ade6-58eb8c96d623\") " Oct 30 19:43:05.326690 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.326588 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-home" pod "5a868137-eb41-433c-ade6-58eb8c96d623" (UID: "5a868137-eb41-433c-ade6-58eb8c96d623"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:05.326832 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.326797 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-tmp-dir" pod "5a868137-eb41-433c-ade6-58eb8c96d623" (UID: "5a868137-eb41-433c-ade6-58eb8c96d623"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:05.326832 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.326825 4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-model-cache\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.326940 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.326895 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25mxq\" (UniqueName: \"kubernetes.io/projected/471f1171-e71f-4da2-862d-5d232c45d6cc-kube-api-access-25mxq\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.327047 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.326940 4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-tmp-dir\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.327047 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.326956 4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-dshm\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.327047 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.326969 4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-tls-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.327047 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.326982 4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-home\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.327285 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.327054 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-model-cache" pod "5a868137-eb41-433c-ade6-58eb8c96d623" (UID: "5a868137-eb41-433c-ade6-58eb8c96d623"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:05.327285 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.327059 4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-dshm\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.327285 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.327093 4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/586f370d-9916-43a9-ac7d-e77cc3fcb92f-dshm\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.327285 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.327117 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7zsj\" (UniqueName: \"kubernetes.io/projected/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-kube-api-access-p7zsj\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.327285 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.327152 4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/471f1171-e71f-4da2-862d-5d232c45d6cc-home\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.327285 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.327167 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf94x\" (UniqueName: \"kubernetes.io/projected/586f370d-9916-43a9-ac7d-e77cc3fcb92f-kube-api-access-nf94x\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.327285 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.327181 4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-model-cache\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.327285 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.327195 4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/586f370d-9916-43a9-ac7d-e77cc3fcb92f-tls-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.327285 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.327208 4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-home\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.327285 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.327220 4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e-tmp-dir\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.327285 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.327234 4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/471f1171-e71f-4da2-862d-5d232c45d6cc-tls-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.328858 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.328797 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-dshm" pod "5a868137-eb41-433c-ade6-58eb8c96d623" (UID: "5a868137-eb41-433c-ade6-58eb8c96d623"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:05.329376 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.329341 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a868137-eb41-433c-ade6-58eb8c96d623-tls-certs" pod "5a868137-eb41-433c-ade6-58eb8c96d623" (UID: "5a868137-eb41-433c-ade6-58eb8c96d623"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:43:05.329492 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.329468 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a868137-eb41-433c-ade6-58eb8c96d623-kube-api-access-k94hr" pod "5a868137-eb41-433c-ade6-58eb8c96d623" (UID: "5a868137-eb41-433c-ade6-58eb8c96d623"). InnerVolumeSpecName "kube-api-access-k94hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:05.427872 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.427841 4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-dshm\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.427872 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.427870 4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-model-cache\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.427872 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.427879 4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a868137-eb41-433c-ade6-58eb8c96d623-tls-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.428144 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.427889 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k94hr\" (UniqueName: \"kubernetes.io/projected/5a868137-eb41-433c-ade6-58eb8c96d623-kube-api-access-k94hr\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.428144 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.427898 4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a868137-eb41-433c-ade6-58eb8c96d623-tmp-dir\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.963672 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.963640 4531 generic.go:356] "Generic (PLEG): container finished" podID="586f370d-9916-43a9-ac7d-e77cc3fcb92f" containerID="3cbc4c5ffbf8572bf9fdff80aa73ebaca28dc4ea0934d2aebe78331cf02556dc" exitCode=0 Oct 30 19:43:05.964230 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.963698 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" Oct 30 19:43:05.964230 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.963726 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" event={"ID":"586f370d-9916-43a9-ac7d-e77cc3fcb92f","Type":"ContainerDied","Data":"3cbc4c5ffbf8572bf9fdff80aa73ebaca28dc4ea0934d2aebe78331cf02556dc"} Oct 30 19:43:05.964230 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.963762 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk" event={"ID":"586f370d-9916-43a9-ac7d-e77cc3fcb92f","Type":"ContainerDied","Data":"7ce9530da13fa1082019675f6066946fd4aeeee91dad6a2bd45b27c0f85ae516"} Oct 30 19:43:05.964230 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.963780 4531 scope.go:122] "RemoveContainer" containerID="3cbc4c5ffbf8572bf9fdff80aa73ebaca28dc4ea0934d2aebe78331cf02556dc" Oct 30 19:43:05.965279 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.965239 4531 generic.go:356] "Generic (PLEG): container finished" podID="5a868137-eb41-433c-ade6-58eb8c96d623" containerID="ffaf31cc4895a3a2b0e24725687a341fa9c99a439b7600d812b62278b55b5e9e" exitCode=0 Oct 30 19:43:05.965405 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.965282 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" Oct 30 19:43:05.965405 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.965292 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" event={"ID":"5a868137-eb41-433c-ade6-58eb8c96d623","Type":"ContainerDied","Data":"ffaf31cc4895a3a2b0e24725687a341fa9c99a439b7600d812b62278b55b5e9e"} Oct 30 19:43:05.965405 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.965334 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z" event={"ID":"5a868137-eb41-433c-ade6-58eb8c96d623","Type":"ContainerDied","Data":"c29348a1962bddecdda97eee0ad284c4d549182d01d0732aa6fa2d97e3b27854"} Oct 30 19:43:05.966851 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.966825 4531 generic.go:356] "Generic (PLEG): container finished" podID="471f1171-e71f-4da2-862d-5d232c45d6cc" containerID="0eacf94a6e4785a277442cd14bbb868e2264ddd5e4916191dfa5fff8b523c053" exitCode=0 Oct 30 19:43:05.966953 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.966930 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" Oct 30 19:43:05.966953 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.966946 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" event={"ID":"471f1171-e71f-4da2-862d-5d232c45d6cc","Type":"ContainerDied","Data":"0eacf94a6e4785a277442cd14bbb868e2264ddd5e4916191dfa5fff8b523c053"} Oct 30 19:43:05.967146 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.966981 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx" event={"ID":"471f1171-e71f-4da2-862d-5d232c45d6cc","Type":"ContainerDied","Data":"01f48d3942f7d3d50d133e67fc4e43909c498f937bfb0348e903d8791669b053"} Oct 30 19:43:05.968467 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.968436 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" Oct 30 19:43:05.968591 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.968474 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn" event={"ID":"9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e","Type":"ContainerDied","Data":"fed1865ad8d83b9935356521d7acc0446c2074565fccb3221caeb24b34cfdd6a"} Oct 30 19:43:05.978588 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.978563 4531 scope.go:122] "RemoveContainer" containerID="3cbc4c5ffbf8572bf9fdff80aa73ebaca28dc4ea0934d2aebe78331cf02556dc" Oct 30 19:43:05.978971 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:43:05.978948 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cbc4c5ffbf8572bf9fdff80aa73ebaca28dc4ea0934d2aebe78331cf02556dc\": container with ID starting with 3cbc4c5ffbf8572bf9fdff80aa73ebaca28dc4ea0934d2aebe78331cf02556dc not found: ID does not exist" containerID="3cbc4c5ffbf8572bf9fdff80aa73ebaca28dc4ea0934d2aebe78331cf02556dc" Oct 30 19:43:05.979106 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.978986 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cbc4c5ffbf8572bf9fdff80aa73ebaca28dc4ea0934d2aebe78331cf02556dc"} err="failed to get container status \"3cbc4c5ffbf8572bf9fdff80aa73ebaca28dc4ea0934d2aebe78331cf02556dc\": rpc error: code = NotFound desc = could not find container \"3cbc4c5ffbf8572bf9fdff80aa73ebaca28dc4ea0934d2aebe78331cf02556dc\": container with ID starting with 3cbc4c5ffbf8572bf9fdff80aa73ebaca28dc4ea0934d2aebe78331cf02556dc not found: ID does not exist" Oct 30 19:43:05.979106 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.979011 4531 scope.go:122] "RemoveContainer" containerID="ffaf31cc4895a3a2b0e24725687a341fa9c99a439b7600d812b62278b55b5e9e" Oct 30 19:43:05.989015 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.988990 4531 scope.go:122] "RemoveContainer" containerID="ffaf31cc4895a3a2b0e24725687a341fa9c99a439b7600d812b62278b55b5e9e" Oct 30 19:43:05.989389 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:43:05.989353 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffaf31cc4895a3a2b0e24725687a341fa9c99a439b7600d812b62278b55b5e9e\": container with ID starting with ffaf31cc4895a3a2b0e24725687a341fa9c99a439b7600d812b62278b55b5e9e not found: ID does not exist" containerID="ffaf31cc4895a3a2b0e24725687a341fa9c99a439b7600d812b62278b55b5e9e" Oct 30 19:43:05.989480 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.989398 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffaf31cc4895a3a2b0e24725687a341fa9c99a439b7600d812b62278b55b5e9e"} err="failed to get container status \"ffaf31cc4895a3a2b0e24725687a341fa9c99a439b7600d812b62278b55b5e9e\": rpc error: code = NotFound desc = could not find container \"ffaf31cc4895a3a2b0e24725687a341fa9c99a439b7600d812b62278b55b5e9e\": container with ID starting with ffaf31cc4895a3a2b0e24725687a341fa9c99a439b7600d812b62278b55b5e9e not found: ID does not exist" Oct 30 19:43:05.989480 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:05.989417 4531 scope.go:122] "RemoveContainer" containerID="0eacf94a6e4785a277442cd14bbb868e2264ddd5e4916191dfa5fff8b523c053" Oct 30 19:43:06.000041 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:06.000016 4531 scope.go:122] "RemoveContainer" containerID="0eacf94a6e4785a277442cd14bbb868e2264ddd5e4916191dfa5fff8b523c053" Oct 30 19:43:06.000621 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:43:06.000585 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eacf94a6e4785a277442cd14bbb868e2264ddd5e4916191dfa5fff8b523c053\": container with ID starting with 0eacf94a6e4785a277442cd14bbb868e2264ddd5e4916191dfa5fff8b523c053 not found: ID does not exist" containerID="0eacf94a6e4785a277442cd14bbb868e2264ddd5e4916191dfa5fff8b523c053" Oct 30 19:43:06.000757 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:06.000634 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eacf94a6e4785a277442cd14bbb868e2264ddd5e4916191dfa5fff8b523c053"} err="failed to get container status \"0eacf94a6e4785a277442cd14bbb868e2264ddd5e4916191dfa5fff8b523c053\": rpc error: code = NotFound desc = could not find container \"0eacf94a6e4785a277442cd14bbb868e2264ddd5e4916191dfa5fff8b523c053\": container with ID starting with 0eacf94a6e4785a277442cd14bbb868e2264ddd5e4916191dfa5fff8b523c053 not found: ID does not exist" Oct 30 19:43:06.000757 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:06.000665 4531 scope.go:122] "RemoveContainer" containerID="db5277d35bda9ea46c09e1c3dadcca354b37eff8d59a82ace86ec474d2e42c66" Oct 30 19:43:06.117894 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:06.117863 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53debdd7-wkdwc" Oct 30 19:43:06.129492 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:06.129450 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk"] Oct 30 19:43:06.133677 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:06.133649 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/simulator-w0-2ade09-kserve-6d4d48958d-n9tqk"] Oct 30 19:43:06.234692 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:06.234608 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b7c30b8-fc0a-43be-81d7-23b089c6881e-kube-api-access-kc8mx\" (UniqueName: \"kubernetes.io/projected/1b7c30b8-fc0a-43be-81d7-23b089c6881e-kube-api-access-kc8mx\") pod \"1b7c30b8-fc0a-43be-81d7-23b089c6881e\" (UID: \"1b7c30b8-fc0a-43be-81d7-23b089c6881e\") " Oct 30 19:43:06.234857 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:06.234712 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b7c30b8-fc0a-43be-81d7-23b089c6881e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7c30b8-fc0a-43be-81d7-23b089c6881e-maas-api-xks-ca\") pod \"1b7c30b8-fc0a-43be-81d7-23b089c6881e\" (UID: \"1b7c30b8-fc0a-43be-81d7-23b089c6881e\") " Oct 30 19:43:06.234857 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:06.234746 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b7c30b8-fc0a-43be-81d7-23b089c6881e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7c30b8-fc0a-43be-81d7-23b089c6881e-maas-api-openshift-ca\") pod \"1b7c30b8-fc0a-43be-81d7-23b089c6881e\" (UID: \"1b7c30b8-fc0a-43be-81d7-23b089c6881e\") " Oct 30 19:43:06.235206 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:06.235181 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b7c30b8-fc0a-43be-81d7-23b089c6881e-maas-api-openshift-ca" pod "1b7c30b8-fc0a-43be-81d7-23b089c6881e" (UID: "1b7c30b8-fc0a-43be-81d7-23b089c6881e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:06.236933 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:06.236908 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7c30b8-fc0a-43be-81d7-23b089c6881e-kube-api-access-kc8mx" pod "1b7c30b8-fc0a-43be-81d7-23b089c6881e" (UID: "1b7c30b8-fc0a-43be-81d7-23b089c6881e"). InnerVolumeSpecName "kube-api-access-kc8mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:06.237041 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:06.236950 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b7c30b8-fc0a-43be-81d7-23b089c6881e-maas-api-xks-ca" pod "1b7c30b8-fc0a-43be-81d7-23b089c6881e" (UID: "1b7c30b8-fc0a-43be-81d7-23b089c6881e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:43:06.242329 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:06.242297 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx"] Oct 30 19:43:06.248081 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:06.248052 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/distinct-w0-2ade09-kserve-7dfdc469b8-dh5rx"] Oct 30 19:43:06.335960 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:06.335919 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7c30b8-fc0a-43be-81d7-23b089c6881e-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:06.335960 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:06.335952 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7c30b8-fc0a-43be-81d7-23b089c6881e-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:06.335960 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:06.335967 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc8mx\" (UniqueName: \"kubernetes.io/projected/1b7c30b8-fc0a-43be-81d7-23b089c6881e-kube-api-access-kc8mx\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:06.398758 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:06.398719 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z"] Oct 30 19:43:06.449355 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:06.449314 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/premium-w0-2ade09-kserve-645d756b69-92k4z"] Oct 30 19:43:06.557350 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:06.557317 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn"] Oct 30 19:43:06.561917 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:06.561894 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2ade09/distinct-2-w0-2ade09-kserve-5679cfd7bc-ng2bn"] Oct 30 19:43:06.978957 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:06.978873 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53debdd7-wkdwc" Oct 30 19:43:06.979481 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:06.979445 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53debdd7-wkdwc" event={"ID":"1b7c30b8-fc0a-43be-81d7-23b089c6881e","Type":"ContainerDied","Data":"5438f76df52083764881c325c4080d5f0d79d0c66d334bf385df7167f7c57f9f"} Oct 30 19:43:06.979620 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:06.979505 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5438f76df52083764881c325c4080d5f0d79d0c66d334bf385df7167f7c57f9f" Oct 30 19:43:08.096770 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:08.096731 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="471f1171-e71f-4da2-862d-5d232c45d6cc" path="/var/lib/kubelet/pods/471f1171-e71f-4da2-862d-5d232c45d6cc/volumes" Oct 30 19:43:08.097176 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:08.097123 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="586f370d-9916-43a9-ac7d-e77cc3fcb92f" path="/var/lib/kubelet/pods/586f370d-9916-43a9-ac7d-e77cc3fcb92f/volumes" Oct 30 19:43:08.097508 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:08.097495 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a868137-eb41-433c-ade6-58eb8c96d623" path="/var/lib/kubelet/pods/5a868137-eb41-433c-ade6-58eb8c96d623/volumes" Oct 30 19:43:08.098246 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:08.098229 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e" path="/var/lib/kubelet/pods/9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e/volumes" Oct 30 19:43:24.661649 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:24.661611 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6"] Oct 30 19:43:24.710591 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:24.710551 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fd61e3-lhkz6"] Oct 30 19:43:25.971599 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:25.971556 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7"] Oct 30 19:43:26.023731 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:26.023684 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2ade09-c9bd7"] Oct 30 19:43:26.096350 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:26.096315 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60a8ea62-3c75-4ddb-b63f-dd880ea8dca1" path="/var/lib/kubelet/pods/60a8ea62-3c75-4ddb-b63f-dd880ea8dca1/volumes" Oct 30 19:43:26.096709 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:26.096697 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3ad1db0-bade-4eb9-9033-abdee1536bde" path="/var/lib/kubelet/pods/b3ad1db0-bade-4eb9-9033-abdee1536bde/volumes" Oct 30 19:43:26.400478 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:43:26.400438 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:43:33.885141 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:33.885094 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb"] Oct 30 19:43:33.885603 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:33.885429 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" podUID="ed1aa9fc-8605-4798-8b39-b75f2dde0337" containerName="istio-proxy" containerID="cri-o://4c43f6c64c0fc366396e836fe1c8cddbc85d7fef47b28b061a7fc186bb9fe5f5" gracePeriod=30 Oct 30 19:43:34.364206 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:34.364162 4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:43:34.364424 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:34.364246 4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" podUID="ed1aa9fc-8605-4798-8b39-b75f2dde0337" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:43:39.134191 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.134163 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 19:43:39.146348 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.146322 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-data\") pod \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " Oct 30 19:43:39.146489 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.146360 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed1aa9fc-8605-4798-8b39-b75f2dde0337-kube-api-access-qhz8k\" (UniqueName: \"kubernetes.io/projected/ed1aa9fc-8605-4798-8b39-b75f2dde0337-kube-api-access-qhz8k\") pod \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " Oct 30 19:43:39.146489 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.146386 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-workload-certs\") pod \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " Oct 30 19:43:39.146489 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.146447 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-workload-socket\") pod \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " Oct 30 19:43:39.146489 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.146474 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-token\" (UniqueName: \"kubernetes.io/projected/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-token\") pod \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " Oct 30 19:43:39.146701 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.146507 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-envoy\") pod \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " Oct 30 19:43:39.146701 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.146603 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-credential-socket\") pod \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " Oct 30 19:43:39.146701 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.146654 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-podinfo\") pod \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " Oct 30 19:43:39.146839 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.146651 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-data" pod "ed1aa9fc-8605-4798-8b39-b75f2dde0337" (UID: "ed1aa9fc-8605-4798-8b39-b75f2dde0337"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:39.146839 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.146790 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istiod-ca-cert\") pod \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\" (UID: \"ed1aa9fc-8605-4798-8b39-b75f2dde0337\") " Oct 30 19:43:39.146839 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.146708 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-workload-certs" pod "ed1aa9fc-8605-4798-8b39-b75f2dde0337" (UID: "ed1aa9fc-8605-4798-8b39-b75f2dde0337"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:39.146839 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.146768 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-credential-socket" pod "ed1aa9fc-8605-4798-8b39-b75f2dde0337" (UID: "ed1aa9fc-8605-4798-8b39-b75f2dde0337"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:39.147060 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.146904 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-workload-socket" pod "ed1aa9fc-8605-4798-8b39-b75f2dde0337" (UID: "ed1aa9fc-8605-4798-8b39-b75f2dde0337"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:39.147166 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.147148 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:39.147240 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.147172 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:39.147240 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.147184 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:39.147240 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.147192 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:39.147365 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.147242 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istiod-ca-cert" pod "ed1aa9fc-8605-4798-8b39-b75f2dde0337" (UID: "ed1aa9fc-8605-4798-8b39-b75f2dde0337"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:39.148900 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.148846 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed1aa9fc-8605-4798-8b39-b75f2dde0337-kube-api-access-qhz8k" pod "ed1aa9fc-8605-4798-8b39-b75f2dde0337" (UID: "ed1aa9fc-8605-4798-8b39-b75f2dde0337"). InnerVolumeSpecName "kube-api-access-qhz8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:39.149103 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.149076 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-podinfo" pod "ed1aa9fc-8605-4798-8b39-b75f2dde0337" (UID: "ed1aa9fc-8605-4798-8b39-b75f2dde0337"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:43:39.149395 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.149370 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-envoy" pod "ed1aa9fc-8605-4798-8b39-b75f2dde0337" (UID: "ed1aa9fc-8605-4798-8b39-b75f2dde0337"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:39.149480 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.149377 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-token" pod "ed1aa9fc-8605-4798-8b39-b75f2dde0337" (UID: "ed1aa9fc-8605-4798-8b39-b75f2dde0337"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:39.170103 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.170071 4531 generic.go:356] "Generic (PLEG): container finished" podID="ed1aa9fc-8605-4798-8b39-b75f2dde0337" containerID="4c43f6c64c0fc366396e836fe1c8cddbc85d7fef47b28b061a7fc186bb9fe5f5" exitCode=0 Oct 30 19:43:39.170318 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.170160 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" event={"ID":"ed1aa9fc-8605-4798-8b39-b75f2dde0337","Type":"ContainerDied","Data":"4c43f6c64c0fc366396e836fe1c8cddbc85d7fef47b28b061a7fc186bb9fe5f5"} Oct 30 19:43:39.170318 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.170172 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" Oct 30 19:43:39.170318 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.170244 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb" event={"ID":"ed1aa9fc-8605-4798-8b39-b75f2dde0337","Type":"ContainerDied","Data":"4881d795d1ca056b213284a1178d4583d4811b0d11c08afd1e024c18b83a5d12"} Oct 30 19:43:39.170318 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.170269 4531 scope.go:122] "RemoveContainer" containerID="4c43f6c64c0fc366396e836fe1c8cddbc85d7fef47b28b061a7fc186bb9fe5f5" Oct 30 19:43:39.180009 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.179991 4531 scope.go:122] "RemoveContainer" containerID="4c43f6c64c0fc366396e836fe1c8cddbc85d7fef47b28b061a7fc186bb9fe5f5" Oct 30 19:43:39.180378 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:43:39.180354 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c43f6c64c0fc366396e836fe1c8cddbc85d7fef47b28b061a7fc186bb9fe5f5\": container with ID starting with 4c43f6c64c0fc366396e836fe1c8cddbc85d7fef47b28b061a7fc186bb9fe5f5 not found: ID does not exist" containerID="4c43f6c64c0fc366396e836fe1c8cddbc85d7fef47b28b061a7fc186bb9fe5f5" Oct 30 19:43:39.180487 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.180389 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c43f6c64c0fc366396e836fe1c8cddbc85d7fef47b28b061a7fc186bb9fe5f5"} err="failed to get container status \"4c43f6c64c0fc366396e836fe1c8cddbc85d7fef47b28b061a7fc186bb9fe5f5\": rpc error: code = NotFound desc = could not find container \"4c43f6c64c0fc366396e836fe1c8cddbc85d7fef47b28b061a7fc186bb9fe5f5\": container with ID starting with 4c43f6c64c0fc366396e836fe1c8cddbc85d7fef47b28b061a7fc186bb9fe5f5 not found: ID does not exist" Oct 30 19:43:39.248028 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.247993 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:39.248028 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.248019 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:39.248028 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.248029 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhz8k\" (UniqueName: \"kubernetes.io/projected/ed1aa9fc-8605-4798-8b39-b75f2dde0337-kube-api-access-qhz8k\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:39.248301 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.248039 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:39.248301 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.248048 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed1aa9fc-8605-4798-8b39-b75f2dde0337-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:43:39.476236 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.476157 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb"] Oct 30 19:43:39.480655 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:39.480626 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2ade09-openshift-default-785b7cf675-pf7vb"] Oct 30 19:43:40.097340 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:43:40.097294 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed1aa9fc-8605-4798-8b39-b75f2dde0337" path="/var/lib/kubelet/pods/ed1aa9fc-8605-4798-8b39-b75f2dde0337/volumes" Oct 30 19:43:56.396977 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:43:56.396937 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:44:03.065271 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:03.065235 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afd486d1-j4s24"] Oct 30 19:44:03.115151 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:03.115091 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afd486d1-j4s24"] Oct 30 19:44:03.116705 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:03.116680 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21979b84-4sckl"] Oct 30 19:44:03.118079 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:03.118055 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21979b84-4sckl"] Oct 30 19:44:03.383914 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:03.383840 4531 scope.go:122] "RemoveContainer" containerID="d37c795f487c10a5071def6abb86995dc2cb3635eb721f7fe4f5094dcff62ab4" Oct 30 19:44:03.392783 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:03.392760 4531 scope.go:122] "RemoveContainer" containerID="0a61a0ca5c04d6050791ccd17ed7b264efff77c105cc15dbfbc2794bbc9d93e3" Oct 30 19:44:03.401040 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:03.401019 4531 scope.go:122] "RemoveContainer" containerID="fa6d67075ebe66e51529ef6f3f2f770dc12fc8cd42aa15f57882a9befa52007e" Oct 30 19:44:03.408964 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:03.408944 4531 scope.go:122] "RemoveContainer" containerID="3918fafdbe1d350982e05c5c76d40aee15c5d626766f79ccd1423f410506e250" Oct 30 19:44:04.097016 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:04.096985 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00856537-9db1-4d00-867d-c409322f77d4" path="/var/lib/kubelet/pods/00856537-9db1-4d00-867d-c409322f77d4/volumes" Oct 30 19:44:04.097436 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:04.097399 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06da2630-8109-40ed-b33d-9051092f06c7" path="/var/lib/kubelet/pods/06da2630-8109-40ed-b33d-9051092f06c7/volumes" Oct 30 19:44:10.096961 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.096930 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-367a2ccf-btznt"] Oct 30 19:44:10.097510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.097273 4531 cpu_manager.go:414] "containerMap: removing container" podUID="9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e" containerName="main" Oct 30 19:44:10.097510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.097287 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e" containerName="main" Oct 30 19:44:10.097510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.097306 4531 cpu_manager.go:414] "containerMap: removing container" podUID="586f370d-9916-43a9-ac7d-e77cc3fcb92f" containerName="main" Oct 30 19:44:10.097510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.097311 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="586f370d-9916-43a9-ac7d-e77cc3fcb92f" containerName="main" Oct 30 19:44:10.097510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.097319 4531 cpu_manager.go:414] "containerMap: removing container" podUID="4b173689-c1d4-419e-bb7e-2ac0afd4f455" containerName="revoke-keys" Oct 30 19:44:10.097510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.097323 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b173689-c1d4-419e-bb7e-2ac0afd4f455" containerName="revoke-keys" Oct 30 19:44:10.097510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.097330 4531 cpu_manager.go:414] "containerMap: removing container" podUID="5bfb7d5b-9b07-48a1-8982-c6d81e47618c" containerName="revoke-keys" Oct 30 19:44:10.097510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.097337 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bfb7d5b-9b07-48a1-8982-c6d81e47618c" containerName="revoke-keys" Oct 30 19:44:10.097510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.097350 4531 cpu_manager.go:414] "containerMap: removing container" podUID="1b7c30b8-fc0a-43be-81d7-23b089c6881e" containerName="revoke-keys" Oct 30 19:44:10.097510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.097358 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b7c30b8-fc0a-43be-81d7-23b089c6881e" containerName="revoke-keys" Oct 30 19:44:10.097510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.097376 4531 cpu_manager.go:414] "containerMap: removing container" podUID="ed1aa9fc-8605-4798-8b39-b75f2dde0337" containerName="istio-proxy" Oct 30 19:44:10.097510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.097380 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed1aa9fc-8605-4798-8b39-b75f2dde0337" containerName="istio-proxy" Oct 30 19:44:10.097510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.097387 4531 cpu_manager.go:414] "containerMap: removing container" podUID="5a868137-eb41-433c-ade6-58eb8c96d623" containerName="main" Oct 30 19:44:10.097510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.097391 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a868137-eb41-433c-ade6-58eb8c96d623" containerName="main" Oct 30 19:44:10.097510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.097398 4531 cpu_manager.go:414] "containerMap: removing container" podUID="471f1171-e71f-4da2-862d-5d232c45d6cc" containerName="main" Oct 30 19:44:10.097510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.097402 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="471f1171-e71f-4da2-862d-5d232c45d6cc" containerName="main" Oct 30 19:44:10.097510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.097457 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dd3f9ea-88e6-4136-9ff9-ad64e6e3104e" containerName="main" Oct 30 19:44:10.097510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.097466 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="586f370d-9916-43a9-ac7d-e77cc3fcb92f" containerName="main" Oct 30 19:44:10.097510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.097484 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b173689-c1d4-419e-bb7e-2ac0afd4f455" containerName="revoke-keys" Oct 30 19:44:10.097510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.097490 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bfb7d5b-9b07-48a1-8982-c6d81e47618c" containerName="revoke-keys" Oct 30 19:44:10.097510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.097496 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b7c30b8-fc0a-43be-81d7-23b089c6881e" containerName="revoke-keys" Oct 30 19:44:10.097510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.097506 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed1aa9fc-8605-4798-8b39-b75f2dde0337" containerName="istio-proxy" Oct 30 19:44:10.097510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.097514 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a868137-eb41-433c-ade6-58eb8c96d623" containerName="main" Oct 30 19:44:10.097510 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.097521 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="471f1171-e71f-4da2-862d-5d232c45d6cc" containerName="main" Oct 30 19:44:10.100634 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.100615 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a2ccf-btznt" Oct 30 19:44:10.205820 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.205774 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:44:10.205820 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.205792 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:44:10.206061 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.205795 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:44:10.206061 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.205897 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:44:10.207638 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.207618 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-367a2ccf-btznt"] Oct 30 19:44:10.232467 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.232430 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134036de-664a-4157-92f5-85dd07e8e8a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-367a2ccf-btznt\" (UID: \"134036de-664a-4157-92f5-85dd07e8e8a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a2ccf-btznt" Oct 30 19:44:10.232467 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.232468 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t6dg\" (UniqueName: \"kubernetes.io/projected/134036de-664a-4157-92f5-85dd07e8e8a3-kube-api-access-6t6dg\") pod \"maas-api-revoke-sub-367a2ccf-btznt\" (UID: \"134036de-664a-4157-92f5-85dd07e8e8a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a2ccf-btznt" Oct 30 19:44:10.232725 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.232490 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134036de-664a-4157-92f5-85dd07e8e8a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-367a2ccf-btznt\" (UID: \"134036de-664a-4157-92f5-85dd07e8e8a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a2ccf-btznt" Oct 30 19:44:10.333753 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.333703 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134036de-664a-4157-92f5-85dd07e8e8a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-367a2ccf-btznt\" (UID: \"134036de-664a-4157-92f5-85dd07e8e8a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a2ccf-btznt" Oct 30 19:44:10.333753 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.333748 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t6dg\" (UniqueName: \"kubernetes.io/projected/134036de-664a-4157-92f5-85dd07e8e8a3-kube-api-access-6t6dg\") pod \"maas-api-revoke-sub-367a2ccf-btznt\" (UID: \"134036de-664a-4157-92f5-85dd07e8e8a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a2ccf-btznt" Oct 30 19:44:10.333992 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.333771 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134036de-664a-4157-92f5-85dd07e8e8a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-367a2ccf-btznt\" (UID: \"134036de-664a-4157-92f5-85dd07e8e8a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a2ccf-btznt" Oct 30 19:44:10.334988 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.334953 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134036de-664a-4157-92f5-85dd07e8e8a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-367a2ccf-btznt\" (UID: \"134036de-664a-4157-92f5-85dd07e8e8a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a2ccf-btznt" Oct 30 19:44:10.337595 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.337562 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134036de-664a-4157-92f5-85dd07e8e8a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-367a2ccf-btznt\" (UID: \"134036de-664a-4157-92f5-85dd07e8e8a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a2ccf-btznt" Oct 30 19:44:10.396180 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.396069 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t6dg\" (UniqueName: \"kubernetes.io/projected/134036de-664a-4157-92f5-85dd07e8e8a3-kube-api-access-6t6dg\") pod \"maas-api-revoke-sub-367a2ccf-btznt\" (UID: \"134036de-664a-4157-92f5-85dd07e8e8a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a2ccf-btznt" Oct 30 19:44:10.413017 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.412982 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a2ccf-btznt" Oct 30 19:44:10.816928 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:10.816889 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-367a2ccf-btznt"] Oct 30 19:44:10.819872 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:44:10.819839 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod134036de_664a_4157_92f5_85dd07e8e8a3.slice/crio-ef495fb43a96c1e190a5e3463d5c417871babcc8d1eea8457890f367e633c56c WatchSource:0}: Error finding container ef495fb43a96c1e190a5e3463d5c417871babcc8d1eea8457890f367e633c56c: Status 404 returned error can't find the container with id ef495fb43a96c1e190a5e3463d5c417871babcc8d1eea8457890f367e633c56c Oct 30 19:44:11.338893 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:11.338855 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a2ccf-btznt" event={"ID":"134036de-664a-4157-92f5-85dd07e8e8a3","Type":"ContainerStarted","Data":"17e1d2d39e1fbaa2ac667af80e93b8ee096dc0d697117520c99dc7e1c7d379f2"} Oct 30 19:44:11.339409 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:11.338909 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a2ccf-btznt" event={"ID":"134036de-664a-4157-92f5-85dd07e8e8a3","Type":"ContainerStarted","Data":"ef495fb43a96c1e190a5e3463d5c417871babcc8d1eea8457890f367e633c56c"} Oct 30 19:44:11.454983 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:11.454863 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a2ccf-btznt" podStartSLOduration=1.454847182 podStartE2EDuration="1.454847182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:11.453396635 +0000 UTC m=+4211.480501595" watchObservedRunningTime="2026-10-30 19:44:11.454847182 +0000 UTC m=+4211.481952141" Oct 30 19:44:12.345207 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:12.345164 4531 generic.go:356] "Generic (PLEG): container finished" podID="134036de-664a-4157-92f5-85dd07e8e8a3" containerID="17e1d2d39e1fbaa2ac667af80e93b8ee096dc0d697117520c99dc7e1c7d379f2" exitCode=0 Oct 30 19:44:12.345660 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:12.345239 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a2ccf-btznt" event={"ID":"134036de-664a-4157-92f5-85dd07e8e8a3","Type":"ContainerDied","Data":"17e1d2d39e1fbaa2ac667af80e93b8ee096dc0d697117520c99dc7e1c7d379f2"} Oct 30 19:44:13.491223 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:13.491192 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a2ccf-btznt" Oct 30 19:44:13.664280 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:13.664183 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/134036de-664a-4157-92f5-85dd07e8e8a3-kube-api-access-6t6dg\" (UniqueName: \"kubernetes.io/projected/134036de-664a-4157-92f5-85dd07e8e8a3-kube-api-access-6t6dg\") pod \"134036de-664a-4157-92f5-85dd07e8e8a3\" (UID: \"134036de-664a-4157-92f5-85dd07e8e8a3\") " Oct 30 19:44:13.664457 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:13.664301 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/134036de-664a-4157-92f5-85dd07e8e8a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134036de-664a-4157-92f5-85dd07e8e8a3-maas-api-openshift-ca\") pod \"134036de-664a-4157-92f5-85dd07e8e8a3\" (UID: \"134036de-664a-4157-92f5-85dd07e8e8a3\") " Oct 30 19:44:13.664457 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:13.664333 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/134036de-664a-4157-92f5-85dd07e8e8a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134036de-664a-4157-92f5-85dd07e8e8a3-maas-api-xks-ca\") pod \"134036de-664a-4157-92f5-85dd07e8e8a3\" (UID: \"134036de-664a-4157-92f5-85dd07e8e8a3\") " Oct 30 19:44:13.664743 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:13.664716 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/134036de-664a-4157-92f5-85dd07e8e8a3-maas-api-openshift-ca" pod "134036de-664a-4157-92f5-85dd07e8e8a3" (UID: "134036de-664a-4157-92f5-85dd07e8e8a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:13.666587 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:13.666564 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/134036de-664a-4157-92f5-85dd07e8e8a3-maas-api-xks-ca" pod "134036de-664a-4157-92f5-85dd07e8e8a3" (UID: "134036de-664a-4157-92f5-85dd07e8e8a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:13.666587 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:13.666567 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/134036de-664a-4157-92f5-85dd07e8e8a3-kube-api-access-6t6dg" pod "134036de-664a-4157-92f5-85dd07e8e8a3" (UID: "134036de-664a-4157-92f5-85dd07e8e8a3"). InnerVolumeSpecName "kube-api-access-6t6dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:13.765052 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:13.765013 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t6dg\" (UniqueName: \"kubernetes.io/projected/134036de-664a-4157-92f5-85dd07e8e8a3-kube-api-access-6t6dg\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:44:13.765052 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:13.765047 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134036de-664a-4157-92f5-85dd07e8e8a3-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:44:13.765052 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:13.765058 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134036de-664a-4157-92f5-85dd07e8e8a3-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:44:14.357801 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:14.357709 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a2ccf-btznt" Oct 30 19:44:14.357801 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:14.357740 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a2ccf-btznt" event={"ID":"134036de-664a-4157-92f5-85dd07e8e8a3","Type":"ContainerDied","Data":"ef495fb43a96c1e190a5e3463d5c417871babcc8d1eea8457890f367e633c56c"} Oct 30 19:44:14.357801 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:14.357788 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef495fb43a96c1e190a5e3463d5c417871babcc8d1eea8457890f367e633c56c" Oct 30 19:44:26.397812 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:44:26.397775 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:44:37.721189 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:37.721157 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb"] Oct 30 19:44:37.721578 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:37.721494 4531 cpu_manager.go:414] "containerMap: removing container" podUID="134036de-664a-4157-92f5-85dd07e8e8a3" containerName="revoke-keys" Oct 30 19:44:37.721578 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:37.721505 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="134036de-664a-4157-92f5-85dd07e8e8a3" containerName="revoke-keys" Oct 30 19:44:37.721578 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:37.721574 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="134036de-664a-4157-92f5-85dd07e8e8a3" containerName="revoke-keys" Oct 30 19:44:37.724860 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:37.724842 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb" Oct 30 19:44:37.774844 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:37.774820 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27872520-a156-4dbd-8d9a-94b00e4e2d4a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb\" (UID: \"27872520-a156-4dbd-8d9a-94b00e4e2d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb" Oct 30 19:44:37.774968 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:37.774867 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h2vn\" (UniqueName: \"kubernetes.io/projected/27872520-a156-4dbd-8d9a-94b00e4e2d4a-kube-api-access-6h2vn\") pod \"maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb\" (UID: \"27872520-a156-4dbd-8d9a-94b00e4e2d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb" Oct 30 19:44:37.774968 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:37.774895 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27872520-a156-4dbd-8d9a-94b00e4e2d4a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb\" (UID: \"27872520-a156-4dbd-8d9a-94b00e4e2d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb" Oct 30 19:44:37.777438 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:37.777410 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:44:37.826705 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:37.826676 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:44:37.826705 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:37.826695 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:44:37.826705 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:37.826694 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:44:37.828387 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:37.828369 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb"] Oct 30 19:44:37.876393 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:37.876365 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27872520-a156-4dbd-8d9a-94b00e4e2d4a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb\" (UID: \"27872520-a156-4dbd-8d9a-94b00e4e2d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb" Oct 30 19:44:37.876546 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:37.876407 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h2vn\" (UniqueName: \"kubernetes.io/projected/27872520-a156-4dbd-8d9a-94b00e4e2d4a-kube-api-access-6h2vn\") pod \"maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb\" (UID: \"27872520-a156-4dbd-8d9a-94b00e4e2d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb" Oct 30 19:44:37.876546 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:37.876430 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27872520-a156-4dbd-8d9a-94b00e4e2d4a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb\" (UID: \"27872520-a156-4dbd-8d9a-94b00e4e2d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb" Oct 30 19:44:37.877021 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:37.876998 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27872520-a156-4dbd-8d9a-94b00e4e2d4a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb\" (UID: \"27872520-a156-4dbd-8d9a-94b00e4e2d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb" Oct 30 19:44:37.878653 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:37.878636 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27872520-a156-4dbd-8d9a-94b00e4e2d4a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb\" (UID: \"27872520-a156-4dbd-8d9a-94b00e4e2d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb" Oct 30 19:44:37.942522 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:37.942500 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h2vn\" (UniqueName: \"kubernetes.io/projected/27872520-a156-4dbd-8d9a-94b00e4e2d4a-kube-api-access-6h2vn\") pod \"maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb\" (UID: \"27872520-a156-4dbd-8d9a-94b00e4e2d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb" Oct 30 19:44:38.037620 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:38.037596 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb" Oct 30 19:44:38.407234 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:38.407208 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb"] Oct 30 19:44:38.410016 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:44:38.409991 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27872520_a156_4dbd_8d9a_94b00e4e2d4a.slice/crio-558990a22a38f8e89aefae1c5ae528918123439c819581b6a57a191b58aefb52 WatchSource:0}: Error finding container 558990a22a38f8e89aefae1c5ae528918123439c819581b6a57a191b58aefb52: Status 404 returned error can't find the container with id 558990a22a38f8e89aefae1c5ae528918123439c819581b6a57a191b58aefb52 Oct 30 19:44:38.492414 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:38.492379 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb" event={"ID":"27872520-a156-4dbd-8d9a-94b00e4e2d4a","Type":"ContainerStarted","Data":"558990a22a38f8e89aefae1c5ae528918123439c819581b6a57a191b58aefb52"} Oct 30 19:44:39.498632 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:39.498600 4531 generic.go:356] "Generic (PLEG): container finished" podID="27872520-a156-4dbd-8d9a-94b00e4e2d4a" containerID="c4016fa38ffff9dfd8ea61b2e812c0488c5b896de6ed1b77b85cd7e963f7eb21" exitCode=0 Oct 30 19:44:39.499002 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:39.498649 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb" event={"ID":"27872520-a156-4dbd-8d9a-94b00e4e2d4a","Type":"ContainerDied","Data":"c4016fa38ffff9dfd8ea61b2e812c0488c5b896de6ed1b77b85cd7e963f7eb21"} Oct 30 19:44:40.637525 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:40.637503 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb" Oct 30 19:44:40.702315 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:40.702289 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27872520-a156-4dbd-8d9a-94b00e4e2d4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27872520-a156-4dbd-8d9a-94b00e4e2d4a-maas-api-xks-ca\") pod \"27872520-a156-4dbd-8d9a-94b00e4e2d4a\" (UID: \"27872520-a156-4dbd-8d9a-94b00e4e2d4a\") " Oct 30 19:44:40.702461 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:40.702352 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27872520-a156-4dbd-8d9a-94b00e4e2d4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27872520-a156-4dbd-8d9a-94b00e4e2d4a-maas-api-openshift-ca\") pod \"27872520-a156-4dbd-8d9a-94b00e4e2d4a\" (UID: \"27872520-a156-4dbd-8d9a-94b00e4e2d4a\") " Oct 30 19:44:40.702461 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:40.702381 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27872520-a156-4dbd-8d9a-94b00e4e2d4a-kube-api-access-6h2vn\" (UniqueName: \"kubernetes.io/projected/27872520-a156-4dbd-8d9a-94b00e4e2d4a-kube-api-access-6h2vn\") pod \"27872520-a156-4dbd-8d9a-94b00e4e2d4a\" (UID: \"27872520-a156-4dbd-8d9a-94b00e4e2d4a\") " Oct 30 19:44:40.702690 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:40.702663 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27872520-a156-4dbd-8d9a-94b00e4e2d4a-maas-api-openshift-ca" pod "27872520-a156-4dbd-8d9a-94b00e4e2d4a" (UID: "27872520-a156-4dbd-8d9a-94b00e4e2d4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:40.704460 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:40.704436 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27872520-a156-4dbd-8d9a-94b00e4e2d4a-maas-api-xks-ca" pod "27872520-a156-4dbd-8d9a-94b00e4e2d4a" (UID: "27872520-a156-4dbd-8d9a-94b00e4e2d4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:40.704642 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:40.704623 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27872520-a156-4dbd-8d9a-94b00e4e2d4a-kube-api-access-6h2vn" pod "27872520-a156-4dbd-8d9a-94b00e4e2d4a" (UID: "27872520-a156-4dbd-8d9a-94b00e4e2d4a"). InnerVolumeSpecName "kube-api-access-6h2vn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:40.803502 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:40.803432 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27872520-a156-4dbd-8d9a-94b00e4e2d4a-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:44:40.803502 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:40.803459 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27872520-a156-4dbd-8d9a-94b00e4e2d4a-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:44:40.803502 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:40.803470 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h2vn\" (UniqueName: \"kubernetes.io/projected/27872520-a156-4dbd-8d9a-94b00e4e2d4a-kube-api-access-6h2vn\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:44:41.509979 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:41.509947 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb" Oct 30 19:44:41.509979 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:41.509960 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb" event={"ID":"27872520-a156-4dbd-8d9a-94b00e4e2d4a","Type":"ContainerDied","Data":"558990a22a38f8e89aefae1c5ae528918123439c819581b6a57a191b58aefb52"} Oct 30 19:44:41.510231 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:41.510002 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="558990a22a38f8e89aefae1c5ae528918123439c819581b6a57a191b58aefb52" Oct 30 19:44:54.257425 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:54.257392 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb"] Oct 30 19:44:54.306396 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:54.306326 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-98b389-m8lqb"] Oct 30 19:44:56.097102 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:44:56.097068 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27872520-a156-4dbd-8d9a-94b00e4e2d4a" path="/var/lib/kubelet/pods/27872520-a156-4dbd-8d9a-94b00e4e2d4a/volumes" Oct 30 19:44:56.396247 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:44:56.396163 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:45:00.186202 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:00.186175 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf"] Oct 30 19:45:00.186673 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:00.186495 4531 cpu_manager.go:414] "containerMap: removing container" podUID="27872520-a156-4dbd-8d9a-94b00e4e2d4a" containerName="revoke-keys" Oct 30 19:45:00.186673 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:00.186506 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27872520-a156-4dbd-8d9a-94b00e4e2d4a" containerName="revoke-keys" Oct 30 19:45:00.186673 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:00.186573 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="27872520-a156-4dbd-8d9a-94b00e4e2d4a" containerName="revoke-keys" Oct 30 19:45:00.190506 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:00.190490 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf" Oct 30 19:45:00.242427 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:00.242403 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:45:00.242427 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:00.242412 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:45:00.275379 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:00.275356 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0508b0cf-7cda-4723-b519-f84c16a29867-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf\" (UID: \"0508b0cf-7cda-4723-b519-f84c16a29867\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf" Oct 30 19:45:00.275505 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:00.275384 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj4gl\" (UniqueName: \"kubernetes.io/projected/0508b0cf-7cda-4723-b519-f84c16a29867-kube-api-access-mj4gl\") pod \"maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf\" (UID: \"0508b0cf-7cda-4723-b519-f84c16a29867\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf" Oct 30 19:45:00.291952 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:00.291915 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:45:00.294147 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:00.294108 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf"] Oct 30 19:45:00.376179 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:00.376149 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0508b0cf-7cda-4723-b519-f84c16a29867-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf\" (UID: \"0508b0cf-7cda-4723-b519-f84c16a29867\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf" Oct 30 19:45:00.376333 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:00.376181 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj4gl\" (UniqueName: \"kubernetes.io/projected/0508b0cf-7cda-4723-b519-f84c16a29867-kube-api-access-mj4gl\") pod \"maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf\" (UID: \"0508b0cf-7cda-4723-b519-f84c16a29867\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf" Oct 30 19:45:00.376752 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:00.376732 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0508b0cf-7cda-4723-b519-f84c16a29867-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf\" (UID: \"0508b0cf-7cda-4723-b519-f84c16a29867\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf" Oct 30 19:45:00.437738 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:00.437682 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj4gl\" (UniqueName: \"kubernetes.io/projected/0508b0cf-7cda-4723-b519-f84c16a29867-kube-api-access-mj4gl\") pod \"maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf\" (UID: \"0508b0cf-7cda-4723-b519-f84c16a29867\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf" Oct 30 19:45:00.503241 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:00.503223 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf" Oct 30 19:45:00.878122 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:00.878096 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf"] Oct 30 19:45:00.880726 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:45:00.880695 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0508b0cf_7cda_4723_b519_f84c16a29867.slice/crio-5d9aabb9b519ff367629fc89f2ccff2e93a0956dbad9d961209d04e326d184c7 WatchSource:0}: Error finding container 5d9aabb9b519ff367629fc89f2ccff2e93a0956dbad9d961209d04e326d184c7: Status 404 returned error can't find the container with id 5d9aabb9b519ff367629fc89f2ccff2e93a0956dbad9d961209d04e326d184c7 Oct 30 19:45:01.619504 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:01.619459 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf" event={"ID":"0508b0cf-7cda-4723-b519-f84c16a29867","Type":"ContainerStarted","Data":"da7abcb0cd5ce5698942b50e00241b17ff5affcb622518a3cb926720643b59f8"} Oct 30 19:45:01.619892 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:01.619534 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf" event={"ID":"0508b0cf-7cda-4723-b519-f84c16a29867","Type":"ContainerStarted","Data":"5d9aabb9b519ff367629fc89f2ccff2e93a0956dbad9d961209d04e326d184c7"} Oct 30 19:45:01.738639 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:01.738575 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf" podStartSLOduration=1.738559068 podStartE2EDuration="1.738559068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:01.738017229 +0000 UTC m=+4261.765122190" watchObservedRunningTime="2026-10-30 19:45:01.738559068 +0000 UTC m=+4261.765664027" Oct 30 19:45:02.626386 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:02.626348 4531 generic.go:356] "Generic (PLEG): container finished" podID="0508b0cf-7cda-4723-b519-f84c16a29867" containerID="da7abcb0cd5ce5698942b50e00241b17ff5affcb622518a3cb926720643b59f8" exitCode=60 Oct 30 19:45:02.626781 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:02.626432 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf" event={"ID":"0508b0cf-7cda-4723-b519-f84c16a29867","Type":"ContainerDied","Data":"da7abcb0cd5ce5698942b50e00241b17ff5affcb622518a3cb926720643b59f8"} Oct 30 19:45:02.626781 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:02.626634 4531 scope.go:122] "RemoveContainer" containerID="da7abcb0cd5ce5698942b50e00241b17ff5affcb622518a3cb926720643b59f8" Oct 30 19:45:03.352488 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:03.352456 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw"] Oct 30 19:45:03.358722 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:03.358684 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw" Oct 30 19:45:03.413413 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:03.413386 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:45:03.415858 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:03.415832 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw"] Oct 30 19:45:03.460895 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:03.460855 4531 scope.go:122] "RemoveContainer" containerID="a12bf608280cacb5171a4c1a24939098f248ff0c5ac9bb46e7a98a930d4182fb" Oct 30 19:45:03.470431 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:03.470413 4531 scope.go:122] "RemoveContainer" containerID="99b9fba606b1e3a8a2ee795d58dd725920986c688c400be19b98740f7b2af7dc" Oct 30 19:45:03.478947 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:03.478930 4531 scope.go:122] "RemoveContainer" containerID="9f2b6cf475d3d2bc24a59726b19dc21a3ee83c68425a01632a736e9ccb3fa290" Oct 30 19:45:03.486398 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:03.486376 4531 scope.go:122] "RemoveContainer" containerID="f166c9d34832d373f1591b5651bfa413a9c0f7447394ace9a6eafafcd4e8bbaf" Oct 30 19:45:03.505634 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:03.505613 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb0a7d8-cdda-4a25-a090-9900dc64377b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw\" (UID: \"0cb0a7d8-cdda-4a25-a090-9900dc64377b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw" Oct 30 19:45:03.505735 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:03.505717 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbz5v\" (UniqueName: \"kubernetes.io/projected/0cb0a7d8-cdda-4a25-a090-9900dc64377b-kube-api-access-xbz5v\") pod \"maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw\" (UID: \"0cb0a7d8-cdda-4a25-a090-9900dc64377b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw" Oct 30 19:45:03.505787 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:03.505776 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb0a7d8-cdda-4a25-a090-9900dc64377b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw\" (UID: \"0cb0a7d8-cdda-4a25-a090-9900dc64377b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw" Oct 30 19:45:03.606432 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:03.606368 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb0a7d8-cdda-4a25-a090-9900dc64377b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw\" (UID: \"0cb0a7d8-cdda-4a25-a090-9900dc64377b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw" Oct 30 19:45:03.606432 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:03.606422 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbz5v\" (UniqueName: \"kubernetes.io/projected/0cb0a7d8-cdda-4a25-a090-9900dc64377b-kube-api-access-xbz5v\") pod \"maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw\" (UID: \"0cb0a7d8-cdda-4a25-a090-9900dc64377b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw" Oct 30 19:45:03.606576 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:03.606450 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb0a7d8-cdda-4a25-a090-9900dc64377b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw\" (UID: \"0cb0a7d8-cdda-4a25-a090-9900dc64377b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw" Oct 30 19:45:03.607103 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:03.607083 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb0a7d8-cdda-4a25-a090-9900dc64377b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw\" (UID: \"0cb0a7d8-cdda-4a25-a090-9900dc64377b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw" Oct 30 19:45:03.608578 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:03.608560 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb0a7d8-cdda-4a25-a090-9900dc64377b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw\" (UID: \"0cb0a7d8-cdda-4a25-a090-9900dc64377b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw" Oct 30 19:45:03.633435 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:03.633411 4531 generic.go:356] "Generic (PLEG): container finished" podID="0508b0cf-7cda-4723-b519-f84c16a29867" containerID="7890f194c49810626c92b4f34df1aa77d42a2fff241dbb95d545a6047fbb6c9d" exitCode=60 Oct 30 19:45:03.633762 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:03.633490 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf" event={"ID":"0508b0cf-7cda-4723-b519-f84c16a29867","Type":"ContainerDied","Data":"7890f194c49810626c92b4f34df1aa77d42a2fff241dbb95d545a6047fbb6c9d"} Oct 30 19:45:03.633762 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:03.633547 4531 scope.go:122] "RemoveContainer" containerID="da7abcb0cd5ce5698942b50e00241b17ff5affcb622518a3cb926720643b59f8" Oct 30 19:45:03.633762 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:03.633720 4531 scope.go:122] "RemoveContainer" containerID="7890f194c49810626c92b4f34df1aa77d42a2fff241dbb95d545a6047fbb6c9d" Oct 30 19:45:03.633954 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:45:03.633916 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf_odh-ai-gateway-infra(0508b0cf-7cda-4723-b519-f84c16a29867)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf" podUID="0508b0cf-7cda-4723-b519-f84c16a29867" Oct 30 19:45:03.667865 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:03.667842 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbz5v\" (UniqueName: \"kubernetes.io/projected/0cb0a7d8-cdda-4a25-a090-9900dc64377b-kube-api-access-xbz5v\") pod \"maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw\" (UID: \"0cb0a7d8-cdda-4a25-a090-9900dc64377b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw" Oct 30 19:45:03.672654 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:03.672636 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw" Oct 30 19:45:04.052899 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:04.052876 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw"] Oct 30 19:45:04.055505 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:45:04.055480 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb0a7d8_cdda_4a25_a090_9900dc64377b.slice/crio-aa8b9dd40733698ffa682136a8187f85386c5cb2a604c330e5568ce385f35836 WatchSource:0}: Error finding container aa8b9dd40733698ffa682136a8187f85386c5cb2a604c330e5568ce385f35836: Status 404 returned error can't find the container with id aa8b9dd40733698ffa682136a8187f85386c5cb2a604c330e5568ce385f35836 Oct 30 19:45:04.640196 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:04.640153 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw" event={"ID":"0cb0a7d8-cdda-4a25-a090-9900dc64377b","Type":"ContainerStarted","Data":"2569d20104fae95c5c53dc56615c7d27335942496fecc6c5b6a1a6aaa22d55cc"} Oct 30 19:45:04.640688 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:04.640222 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw" event={"ID":"0cb0a7d8-cdda-4a25-a090-9900dc64377b","Type":"ContainerStarted","Data":"aa8b9dd40733698ffa682136a8187f85386c5cb2a604c330e5568ce385f35836"} Oct 30 19:45:04.641897 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:04.641872 4531 scope.go:122] "RemoveContainer" containerID="7890f194c49810626c92b4f34df1aa77d42a2fff241dbb95d545a6047fbb6c9d" Oct 30 19:45:04.642060 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:45:04.642043 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf_odh-ai-gateway-infra(0508b0cf-7cda-4723-b519-f84c16a29867)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf" podUID="0508b0cf-7cda-4723-b519-f84c16a29867" Oct 30 19:45:04.782608 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:04.782559 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw" podStartSLOduration=1.7825460720000001 podStartE2EDuration="1.782546072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:04.780528587 +0000 UTC m=+4264.807633556" watchObservedRunningTime="2026-10-30 19:45:04.782546072 +0000 UTC m=+4264.809651031" Oct 30 19:45:05.647850 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:05.647812 4531 generic.go:356] "Generic (PLEG): container finished" podID="0cb0a7d8-cdda-4a25-a090-9900dc64377b" containerID="2569d20104fae95c5c53dc56615c7d27335942496fecc6c5b6a1a6aaa22d55cc" exitCode=0 Oct 30 19:45:05.648321 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:05.647891 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw" event={"ID":"0cb0a7d8-cdda-4a25-a090-9900dc64377b","Type":"ContainerDied","Data":"2569d20104fae95c5c53dc56615c7d27335942496fecc6c5b6a1a6aaa22d55cc"} Oct 30 19:45:06.794156 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:06.794081 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw" Oct 30 19:45:06.935181 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:06.935073 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cb0a7d8-cdda-4a25-a090-9900dc64377b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb0a7d8-cdda-4a25-a090-9900dc64377b-maas-api-openshift-ca\") pod \"0cb0a7d8-cdda-4a25-a090-9900dc64377b\" (UID: \"0cb0a7d8-cdda-4a25-a090-9900dc64377b\") " Oct 30 19:45:06.935350 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:06.935204 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cb0a7d8-cdda-4a25-a090-9900dc64377b-kube-api-access-xbz5v\" (UniqueName: \"kubernetes.io/projected/0cb0a7d8-cdda-4a25-a090-9900dc64377b-kube-api-access-xbz5v\") pod \"0cb0a7d8-cdda-4a25-a090-9900dc64377b\" (UID: \"0cb0a7d8-cdda-4a25-a090-9900dc64377b\") " Oct 30 19:45:06.935350 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:06.935301 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cb0a7d8-cdda-4a25-a090-9900dc64377b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb0a7d8-cdda-4a25-a090-9900dc64377b-maas-api-xks-ca\") pod \"0cb0a7d8-cdda-4a25-a090-9900dc64377b\" (UID: \"0cb0a7d8-cdda-4a25-a090-9900dc64377b\") " Oct 30 19:45:06.935546 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:06.935524 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb0a7d8-cdda-4a25-a090-9900dc64377b-maas-api-openshift-ca" pod "0cb0a7d8-cdda-4a25-a090-9900dc64377b" (UID: "0cb0a7d8-cdda-4a25-a090-9900dc64377b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:06.937542 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:06.937511 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cb0a7d8-cdda-4a25-a090-9900dc64377b-maas-api-xks-ca" pod "0cb0a7d8-cdda-4a25-a090-9900dc64377b" (UID: "0cb0a7d8-cdda-4a25-a090-9900dc64377b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:06.937674 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:06.937573 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb0a7d8-cdda-4a25-a090-9900dc64377b-kube-api-access-xbz5v" pod "0cb0a7d8-cdda-4a25-a090-9900dc64377b" (UID: "0cb0a7d8-cdda-4a25-a090-9900dc64377b"). InnerVolumeSpecName "kube-api-access-xbz5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:07.036356 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:07.036309 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbz5v\" (UniqueName: \"kubernetes.io/projected/0cb0a7d8-cdda-4a25-a090-9900dc64377b-kube-api-access-xbz5v\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:45:07.036356 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:07.036343 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb0a7d8-cdda-4a25-a090-9900dc64377b-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:45:07.036356 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:07.036355 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb0a7d8-cdda-4a25-a090-9900dc64377b-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:45:07.659542 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:07.659501 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw" event={"ID":"0cb0a7d8-cdda-4a25-a090-9900dc64377b","Type":"ContainerDied","Data":"aa8b9dd40733698ffa682136a8187f85386c5cb2a604c330e5568ce385f35836"} Oct 30 19:45:07.659542 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:07.659521 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw" Oct 30 19:45:07.659542 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:07.659551 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa8b9dd40733698ffa682136a8187f85386c5cb2a604c330e5568ce385f35836" Oct 30 19:45:16.092555 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:16.092462 4531 scope.go:122] "RemoveContainer" containerID="7890f194c49810626c92b4f34df1aa77d42a2fff241dbb95d545a6047fbb6c9d" Oct 30 19:45:16.718633 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:16.718532 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf" event={"ID":"0508b0cf-7cda-4723-b519-f84c16a29867","Type":"ContainerStarted","Data":"055ed8ef856accda943efc2e57b7be64707f0f9bd91361ab60330b11ffb76bed"} Oct 30 19:45:17.726245 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:17.726202 4531 generic.go:356] "Generic (PLEG): container finished" podID="0508b0cf-7cda-4723-b519-f84c16a29867" containerID="055ed8ef856accda943efc2e57b7be64707f0f9bd91361ab60330b11ffb76bed" exitCode=60 Oct 30 19:45:17.726770 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:17.726281 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf" event={"ID":"0508b0cf-7cda-4723-b519-f84c16a29867","Type":"ContainerDied","Data":"055ed8ef856accda943efc2e57b7be64707f0f9bd91361ab60330b11ffb76bed"} Oct 30 19:45:17.726770 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:17.726359 4531 scope.go:122] "RemoveContainer" containerID="7890f194c49810626c92b4f34df1aa77d42a2fff241dbb95d545a6047fbb6c9d" Oct 30 19:45:17.726770 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:17.726700 4531 scope.go:122] "RemoveContainer" containerID="055ed8ef856accda943efc2e57b7be64707f0f9bd91361ab60330b11ffb76bed" Oct 30 19:45:17.727010 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:45:17.726987 4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf_odh-ai-gateway-infra(0508b0cf-7cda-4723-b519-f84c16a29867)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf" podUID="0508b0cf-7cda-4723-b519-f84c16a29867" Oct 30 19:45:17.847499 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:17.847456 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf"] Oct 30 19:45:18.882055 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:18.882011 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf" Oct 30 19:45:18.944500 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:18.944462 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0508b0cf-7cda-4723-b519-f84c16a29867-kube-api-access-mj4gl\" (UniqueName: \"kubernetes.io/projected/0508b0cf-7cda-4723-b519-f84c16a29867-kube-api-access-mj4gl\") pod \"0508b0cf-7cda-4723-b519-f84c16a29867\" (UID: \"0508b0cf-7cda-4723-b519-f84c16a29867\") " Oct 30 19:45:18.944713 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:18.944595 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0508b0cf-7cda-4723-b519-f84c16a29867-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0508b0cf-7cda-4723-b519-f84c16a29867-maas-api-service-ca\") pod \"0508b0cf-7cda-4723-b519-f84c16a29867\" (UID: \"0508b0cf-7cda-4723-b519-f84c16a29867\") " Oct 30 19:45:18.944970 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:18.944934 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0508b0cf-7cda-4723-b519-f84c16a29867-maas-api-service-ca" pod "0508b0cf-7cda-4723-b519-f84c16a29867" (UID: "0508b0cf-7cda-4723-b519-f84c16a29867"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:18.946897 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:18.946870 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0508b0cf-7cda-4723-b519-f84c16a29867-kube-api-access-mj4gl" pod "0508b0cf-7cda-4723-b519-f84c16a29867" (UID: "0508b0cf-7cda-4723-b519-f84c16a29867"). InnerVolumeSpecName "kube-api-access-mj4gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:19.045292 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:19.045237 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj4gl\" (UniqueName: \"kubernetes.io/projected/0508b0cf-7cda-4723-b519-f84c16a29867-kube-api-access-mj4gl\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:45:19.045292 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:19.045288 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0508b0cf-7cda-4723-b519-f84c16a29867-maas-api-service-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:45:19.741605 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:19.741556 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf" event={"ID":"0508b0cf-7cda-4723-b519-f84c16a29867","Type":"ContainerDied","Data":"5d9aabb9b519ff367629fc89f2ccff2e93a0956dbad9d961209d04e326d184c7"} Oct 30 19:45:19.741605 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:19.741590 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf" Oct 30 19:45:19.741853 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:19.741654 4531 scope.go:122] "RemoveContainer" containerID="055ed8ef856accda943efc2e57b7be64707f0f9bd91361ab60330b11ffb76bed" Oct 30 19:45:19.907438 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:19.907391 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf"] Oct 30 19:45:19.956779 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:19.956731 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e7929d-29889825-957lf"] Oct 30 19:45:20.098440 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:20.098390 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0508b0cf-7cda-4723-b519-f84c16a29867" path="/var/lib/kubelet/pods/0508b0cf-7cda-4723-b519-f84c16a29867/volumes" Oct 30 19:45:26.399813 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:45:26.399779 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:45:27.350645 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:27.350602 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw"] Oct 30 19:45:27.399779 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:27.399736 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5da1ae-q4kkw"] Oct 30 19:45:28.097644 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:28.097596 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cb0a7d8-cdda-4a25-a090-9900dc64377b" path="/var/lib/kubelet/pods/0cb0a7d8-cdda-4a25-a090-9900dc64377b/volumes" Oct 30 19:45:46.553292 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.553251 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45c2c222-pr97d"] Oct 30 19:45:46.553864 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.553599 4531 cpu_manager.go:414] "containerMap: removing container" podUID="0cb0a7d8-cdda-4a25-a090-9900dc64377b" containerName="revoke-keys" Oct 30 19:45:46.553864 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.553611 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cb0a7d8-cdda-4a25-a090-9900dc64377b" containerName="revoke-keys" Oct 30 19:45:46.553864 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.553639 4531 cpu_manager.go:414] "containerMap: removing container" podUID="0508b0cf-7cda-4723-b519-f84c16a29867" containerName="cleanup" Oct 30 19:45:46.553864 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.553644 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0508b0cf-7cda-4723-b519-f84c16a29867" containerName="cleanup" Oct 30 19:45:46.553864 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.553653 4531 cpu_manager.go:414] "containerMap: removing container" podUID="0508b0cf-7cda-4723-b519-f84c16a29867" containerName="cleanup" Oct 30 19:45:46.553864 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.553658 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0508b0cf-7cda-4723-b519-f84c16a29867" containerName="cleanup" Oct 30 19:45:46.553864 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.553715 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cb0a7d8-cdda-4a25-a090-9900dc64377b" containerName="revoke-keys" Oct 30 19:45:46.553864 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.553729 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0508b0cf-7cda-4723-b519-f84c16a29867" containerName="cleanup" Oct 30 19:45:46.553864 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.553738 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0508b0cf-7cda-4723-b519-f84c16a29867" containerName="cleanup" Oct 30 19:45:46.553864 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.553749 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0508b0cf-7cda-4723-b519-f84c16a29867" containerName="cleanup" Oct 30 19:45:46.556873 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.556845 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c2c222-pr97d" Oct 30 19:45:46.610056 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.610020 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:45:46.610295 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.610158 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:45:46.611437 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.611409 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:45:46.611560 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.611435 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qg7n\"" Oct 30 19:45:46.660704 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.660666 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45c2c222-pr97d"] Oct 30 19:45:46.696336 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.696293 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j49pd\" (UniqueName: \"kubernetes.io/projected/97604a9b-3f63-49d0-8377-4d7a30fa4a08-kube-api-access-j49pd\") pod \"maas-api-revoke-sub-45c2c222-pr97d\" (UID: \"97604a9b-3f63-49d0-8377-4d7a30fa4a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c2c222-pr97d" Oct 30 19:45:46.696540 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.696358 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97604a9b-3f63-49d0-8377-4d7a30fa4a08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45c2c222-pr97d\" (UID: \"97604a9b-3f63-49d0-8377-4d7a30fa4a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c2c222-pr97d" Oct 30 19:45:46.696540 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.696478 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97604a9b-3f63-49d0-8377-4d7a30fa4a08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45c2c222-pr97d\" (UID: \"97604a9b-3f63-49d0-8377-4d7a30fa4a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c2c222-pr97d" Oct 30 19:45:46.797944 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.797903 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97604a9b-3f63-49d0-8377-4d7a30fa4a08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45c2c222-pr97d\" (UID: \"97604a9b-3f63-49d0-8377-4d7a30fa4a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c2c222-pr97d" Oct 30 19:45:46.798153 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.797988 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97604a9b-3f63-49d0-8377-4d7a30fa4a08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45c2c222-pr97d\" (UID: \"97604a9b-3f63-49d0-8377-4d7a30fa4a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c2c222-pr97d" Oct 30 19:45:46.798153 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.798032 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j49pd\" (UniqueName: \"kubernetes.io/projected/97604a9b-3f63-49d0-8377-4d7a30fa4a08-kube-api-access-j49pd\") pod \"maas-api-revoke-sub-45c2c222-pr97d\" (UID: \"97604a9b-3f63-49d0-8377-4d7a30fa4a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c2c222-pr97d" Oct 30 19:45:46.798787 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.798763 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97604a9b-3f63-49d0-8377-4d7a30fa4a08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45c2c222-pr97d\" (UID: \"97604a9b-3f63-49d0-8377-4d7a30fa4a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c2c222-pr97d" Oct 30 19:45:46.800669 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.800642 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97604a9b-3f63-49d0-8377-4d7a30fa4a08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45c2c222-pr97d\" (UID: \"97604a9b-3f63-49d0-8377-4d7a30fa4a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c2c222-pr97d" Oct 30 19:45:46.859755 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.859659 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j49pd\" (UniqueName: \"kubernetes.io/projected/97604a9b-3f63-49d0-8377-4d7a30fa4a08-kube-api-access-j49pd\") pod \"maas-api-revoke-sub-45c2c222-pr97d\" (UID: \"97604a9b-3f63-49d0-8377-4d7a30fa4a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c2c222-pr97d" Oct 30 19:45:46.870489 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:46.870456 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c2c222-pr97d" Oct 30 19:45:47.252959 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:47.252921 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45c2c222-pr97d"] Oct 30 19:45:47.255434 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:45:47.255401 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97604a9b_3f63_49d0_8377_4d7a30fa4a08.slice/crio-cfbbcbd1eb3dd547cdec231f5c394d9489594b2851b5d22ba1369cb226e2a66d WatchSource:0}: Error finding container cfbbcbd1eb3dd547cdec231f5c394d9489594b2851b5d22ba1369cb226e2a66d: Status 404 returned error can't find the container with id cfbbcbd1eb3dd547cdec231f5c394d9489594b2851b5d22ba1369cb226e2a66d Oct 30 19:45:47.896165 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:47.896032 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c2c222-pr97d" event={"ID":"97604a9b-3f63-49d0-8377-4d7a30fa4a08","Type":"ContainerStarted","Data":"676be09a0872c6c1f670d80be1118011c9f090a8275a031cc833a5758e361c7c"} Oct 30 19:45:47.896165 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:47.896114 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c2c222-pr97d" event={"ID":"97604a9b-3f63-49d0-8377-4d7a30fa4a08","Type":"ContainerStarted","Data":"cfbbcbd1eb3dd547cdec231f5c394d9489594b2851b5d22ba1369cb226e2a66d"} Oct 30 19:45:48.019689 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:48.019631 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c2c222-pr97d" podStartSLOduration=2.019611714 podStartE2EDuration="2.019611714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:48.018037352 +0000 UTC m=+4308.045142311" watchObservedRunningTime="2026-10-30 19:45:48.019611714 +0000 UTC m=+4308.046716715" Oct 30 19:45:48.902436 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:48.902399 4531 generic.go:356] "Generic (PLEG): container finished" podID="97604a9b-3f63-49d0-8377-4d7a30fa4a08" containerID="676be09a0872c6c1f670d80be1118011c9f090a8275a031cc833a5758e361c7c" exitCode=0 Oct 30 19:45:48.902853 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:48.902486 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c2c222-pr97d" event={"ID":"97604a9b-3f63-49d0-8377-4d7a30fa4a08","Type":"ContainerDied","Data":"676be09a0872c6c1f670d80be1118011c9f090a8275a031cc833a5758e361c7c"} Oct 30 19:45:50.054536 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:50.054500 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c2c222-pr97d" Oct 30 19:45:50.129751 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:50.129714 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97604a9b-3f63-49d0-8377-4d7a30fa4a08-kube-api-access-j49pd\" (UniqueName: \"kubernetes.io/projected/97604a9b-3f63-49d0-8377-4d7a30fa4a08-kube-api-access-j49pd\") pod \"97604a9b-3f63-49d0-8377-4d7a30fa4a08\" (UID: \"97604a9b-3f63-49d0-8377-4d7a30fa4a08\") " Oct 30 19:45:50.129983 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:50.129831 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97604a9b-3f63-49d0-8377-4d7a30fa4a08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97604a9b-3f63-49d0-8377-4d7a30fa4a08-maas-api-xks-ca\") pod \"97604a9b-3f63-49d0-8377-4d7a30fa4a08\" (UID: \"97604a9b-3f63-49d0-8377-4d7a30fa4a08\") " Oct 30 19:45:50.129983 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:50.129904 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97604a9b-3f63-49d0-8377-4d7a30fa4a08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97604a9b-3f63-49d0-8377-4d7a30fa4a08-maas-api-openshift-ca\") pod \"97604a9b-3f63-49d0-8377-4d7a30fa4a08\" (UID: \"97604a9b-3f63-49d0-8377-4d7a30fa4a08\") " Oct 30 19:45:50.130344 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:50.130320 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97604a9b-3f63-49d0-8377-4d7a30fa4a08-maas-api-openshift-ca" pod "97604a9b-3f63-49d0-8377-4d7a30fa4a08" (UID: "97604a9b-3f63-49d0-8377-4d7a30fa4a08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:50.132187 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:50.132157 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97604a9b-3f63-49d0-8377-4d7a30fa4a08-maas-api-xks-ca" pod "97604a9b-3f63-49d0-8377-4d7a30fa4a08" (UID: "97604a9b-3f63-49d0-8377-4d7a30fa4a08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:50.132331 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:50.132274 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97604a9b-3f63-49d0-8377-4d7a30fa4a08-kube-api-access-j49pd" pod "97604a9b-3f63-49d0-8377-4d7a30fa4a08" (UID: "97604a9b-3f63-49d0-8377-4d7a30fa4a08"). InnerVolumeSpecName "kube-api-access-j49pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:50.231420 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:50.231311 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97604a9b-3f63-49d0-8377-4d7a30fa4a08-maas-api-openshift-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:45:50.231420 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:50.231354 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j49pd\" (UniqueName: \"kubernetes.io/projected/97604a9b-3f63-49d0-8377-4d7a30fa4a08-kube-api-access-j49pd\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:45:50.231420 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:50.231366 4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97604a9b-3f63-49d0-8377-4d7a30fa4a08-maas-api-xks-ca\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:45:50.914635 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:50.914606 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c2c222-pr97d" Oct 30 19:45:50.914832 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:50.914640 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45c2c222-pr97d" event={"ID":"97604a9b-3f63-49d0-8377-4d7a30fa4a08","Type":"ContainerDied","Data":"cfbbcbd1eb3dd547cdec231f5c394d9489594b2851b5d22ba1369cb226e2a66d"} Oct 30 19:45:50.914832 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:45:50.914692 4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfbbcbd1eb3dd547cdec231f5c394d9489594b2851b5d22ba1369cb226e2a66d" Oct 30 19:45:56.399640 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:45:56.399600 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:46:03.518294 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:03.518256 4531 scope.go:122] "RemoveContainer" containerID="4e203e12f839eb16b6b255d466a5ecb1bc42dff3f7ae64edf9a991432609daa8" Oct 30 19:46:03.527754 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:03.527730 4531 scope.go:122] "RemoveContainer" containerID="e65f32bb3870462a6658005e75d0d840280d817b01f388ade36e5a070809104c" Oct 30 19:46:11.028986 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:11.028941 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2"] Oct 30 19:46:11.029475 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:11.029191 4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" podUID="efe5fedb-d8ee-418b-89d6-1d1891e436df" containerName="istio-proxy" containerID="cri-o://92c63cdca82c8b7236eb30ed333fc21a000e9d30d3b8a08a87ea165d0ae45046" gracePeriod=30 Oct 30 19:46:13.358332 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:13.358298 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-rkjnb_0c895121-1801-4d2a-a4d2-c33e167ec4b5/kuadrant-console-plugin/0.log" Oct 30 19:46:16.276432 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.276393 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 19:46:16.377795 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.377697 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-envoy\") pod \"efe5fedb-d8ee-418b-89d6-1d1891e436df\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " Oct 30 19:46:16.377795 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.377756 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-workload-certs\") pod \"efe5fedb-d8ee-418b-89d6-1d1891e436df\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " Oct 30 19:46:16.377795 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.377785 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efe5fedb-d8ee-418b-89d6-1d1891e436df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efe5fedb-d8ee-418b-89d6-1d1891e436df-istiod-ca-cert\") pod \"efe5fedb-d8ee-418b-89d6-1d1891e436df\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " Oct 30 19:46:16.378115 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.377824 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-workload-socket\") pod \"efe5fedb-d8ee-418b-89d6-1d1891e436df\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " Oct 30 19:46:16.378115 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.377849 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efe5fedb-d8ee-418b-89d6-1d1891e436df-kube-api-access-4xfrx\" (UniqueName: \"kubernetes.io/projected/efe5fedb-d8ee-418b-89d6-1d1891e436df-kube-api-access-4xfrx\") pod \"efe5fedb-d8ee-418b-89d6-1d1891e436df\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " Oct 30 19:46:16.378115 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.377879 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-data\") pod \"efe5fedb-d8ee-418b-89d6-1d1891e436df\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " Oct 30 19:46:16.378115 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.377903 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-podinfo\") pod \"efe5fedb-d8ee-418b-89d6-1d1891e436df\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " Oct 30 19:46:16.378115 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.377983 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-credential-socket\") pod \"efe5fedb-d8ee-418b-89d6-1d1891e436df\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " Oct 30 19:46:16.378115 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.378007 4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-token\" (UniqueName: \"kubernetes.io/projected/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-token\") pod \"efe5fedb-d8ee-418b-89d6-1d1891e436df\" (UID: \"efe5fedb-d8ee-418b-89d6-1d1891e436df\") " Oct 30 19:46:16.378115 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.378088 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-workload-certs" pod "efe5fedb-d8ee-418b-89d6-1d1891e436df" (UID: "efe5fedb-d8ee-418b-89d6-1d1891e436df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:16.378560 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.378233 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-workload-socket" pod "efe5fedb-d8ee-418b-89d6-1d1891e436df" (UID: "efe5fedb-d8ee-418b-89d6-1d1891e436df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:16.378560 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.378293 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-data" pod "efe5fedb-d8ee-418b-89d6-1d1891e436df" (UID: "efe5fedb-d8ee-418b-89d6-1d1891e436df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:16.378560 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.378340 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efe5fedb-d8ee-418b-89d6-1d1891e436df-istiod-ca-cert" pod "efe5fedb-d8ee-418b-89d6-1d1891e436df" (UID: "efe5fedb-d8ee-418b-89d6-1d1891e436df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:46:16.378560 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.378421 4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-workload-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.378560 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.378441 4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-data\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.378560 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.378458 4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-workload-certs\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.379055 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.378762 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-credential-socket" pod "efe5fedb-d8ee-418b-89d6-1d1891e436df" (UID: "efe5fedb-d8ee-418b-89d6-1d1891e436df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:16.380194 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.380164 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-envoy" pod "efe5fedb-d8ee-418b-89d6-1d1891e436df" (UID: "efe5fedb-d8ee-418b-89d6-1d1891e436df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:16.380518 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.380486 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe5fedb-d8ee-418b-89d6-1d1891e436df-kube-api-access-4xfrx" pod "efe5fedb-d8ee-418b-89d6-1d1891e436df" (UID: "efe5fedb-d8ee-418b-89d6-1d1891e436df"). InnerVolumeSpecName "kube-api-access-4xfrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:16.380633 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.380611 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-podinfo" pod "efe5fedb-d8ee-418b-89d6-1d1891e436df" (UID: "efe5fedb-d8ee-418b-89d6-1d1891e436df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:46:16.380781 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.380761 4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-token" pod "efe5fedb-d8ee-418b-89d6-1d1891e436df" (UID: "efe5fedb-d8ee-418b-89d6-1d1891e436df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:16.479911 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.479872 4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efe5fedb-d8ee-418b-89d6-1d1891e436df-istiod-ca-cert\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.479911 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.479904 4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xfrx\" (UniqueName: \"kubernetes.io/projected/efe5fedb-d8ee-418b-89d6-1d1891e436df-kube-api-access-4xfrx\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.479911 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.479915 4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-podinfo\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.480214 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.479924 4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-credential-socket\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.480214 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.479934 4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-token\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.480214 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.479942 4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efe5fedb-d8ee-418b-89d6-1d1891e436df-istio-envoy\") on node \"ip-10-0-133-56.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.713819 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:16.713726 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-56jrt_a47b1484-d120-4c8e-ad86-1309eb472e84/manager/0.log" Oct 30 19:46:17.049744 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:17.049708 4531 generic.go:356] "Generic (PLEG): container finished" podID="efe5fedb-d8ee-418b-89d6-1d1891e436df" containerID="92c63cdca82c8b7236eb30ed333fc21a000e9d30d3b8a08a87ea165d0ae45046" exitCode=0 Oct 30 19:46:17.049950 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:17.049762 4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" Oct 30 19:46:17.049950 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:17.049790 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" event={"ID":"efe5fedb-d8ee-418b-89d6-1d1891e436df","Type":"ContainerDied","Data":"92c63cdca82c8b7236eb30ed333fc21a000e9d30d3b8a08a87ea165d0ae45046"} Oct 30 19:46:17.049950 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:17.049842 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2" event={"ID":"efe5fedb-d8ee-418b-89d6-1d1891e436df","Type":"ContainerDied","Data":"c1f5cff59ecd53a6acc9a69a25ff52903d6cf38769b18ea4b3e225ddcc0c3c5d"} Oct 30 19:46:17.049950 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:17.049865 4531 scope.go:122] "RemoveContainer" containerID="92c63cdca82c8b7236eb30ed333fc21a000e9d30d3b8a08a87ea165d0ae45046" Oct 30 19:46:17.059453 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:17.059431 4531 scope.go:122] "RemoveContainer" containerID="92c63cdca82c8b7236eb30ed333fc21a000e9d30d3b8a08a87ea165d0ae45046" Oct 30 19:46:17.059753 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:46:17.059735 4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c63cdca82c8b7236eb30ed333fc21a000e9d30d3b8a08a87ea165d0ae45046\": container with ID starting with 92c63cdca82c8b7236eb30ed333fc21a000e9d30d3b8a08a87ea165d0ae45046 not found: ID does not exist" containerID="92c63cdca82c8b7236eb30ed333fc21a000e9d30d3b8a08a87ea165d0ae45046" Oct 30 19:46:17.059829 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:17.059763 4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c63cdca82c8b7236eb30ed333fc21a000e9d30d3b8a08a87ea165d0ae45046"} err="failed to get container status \"92c63cdca82c8b7236eb30ed333fc21a000e9d30d3b8a08a87ea165d0ae45046\": rpc error: code = NotFound desc = could not find container \"92c63cdca82c8b7236eb30ed333fc21a000e9d30d3b8a08a87ea165d0ae45046\": container with ID starting with 92c63cdca82c8b7236eb30ed333fc21a000e9d30d3b8a08a87ea165d0ae45046 not found: ID does not exist" Oct 30 19:46:17.268170 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:17.268111 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2"] Oct 30 19:46:17.290022 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:17.289990 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e7929d-openshift-default-7f6f56644f-f4fp2"] Oct 30 19:46:18.097529 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:18.097495 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efe5fedb-d8ee-418b-89d6-1d1891e436df" path="/var/lib/kubelet/pods/efe5fedb-d8ee-418b-89d6-1d1891e436df/volumes" Oct 30 19:46:26.398042 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:46:26.398002 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:46:30.482328 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:30.482287 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-56jrt_a47b1484-d120-4c8e-ad86-1309eb472e84/manager/0.log" Oct 30 19:46:33.145188 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:33.145156 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-367a2ccf-btznt_134036de-664a-4157-92f5-85dd07e8e8a3/revoke-keys/0.log" Oct 30 19:46:33.369281 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:33.369244 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-45c2c222-pr97d_97604a9b-3f63-49d0-8377-4d7a30fa4a08/revoke-keys/0.log" Oct 30 19:46:33.588763 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:33.588720 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-53debdd7-wkdwc_1b7c30b8-fc0a-43be-81d7-23b089c6881e/revoke-keys/0.log" Oct 30 19:46:33.811917 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:33.811886 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5c907a8b-5wvmk_71a76f93-3d23-43e6-a3ac-c63938d30f53/revoke-keys/0.log" Oct 30 19:46:34.059563 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:34.059527 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7529a119-chzjh_5bfb7d5b-9b07-48a1-8982-c6d81e47618c/revoke-keys/0.log" Oct 30 19:46:34.518321 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:34.518293 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-81d557ab-q46p4_4b173689-c1d4-419e-bb7e-2ac0afd4f455/revoke-keys/0.log" Oct 30 19:46:34.985400 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:34.985319 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ea583398-jdbxh_cfe5743d-6d47-48a3-b86c-927895ba3caf/revoke-keys/0.log" Oct 30 19:46:37.528496 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:37.528467 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-rkjnb_0c895121-1801-4d2a-a4d2-c33e167ec4b5/kuadrant-console-plugin/0.log" Oct 30 19:46:40.160853 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:40.160818 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9468cdd5d-vhp9x_2650a306-fa87-40d3-b893-5480a959ac51/kube-auth-proxy/0.log" Oct 30 19:46:40.637541 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:46:40.637507 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-6khdd_f67d6e91-06ca-4538-8c0e-40d5421544b6/istio-proxy/0.log" Oct 30 19:46:56.396627 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:46:56.396589 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:47:03.560278 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:03.560245 4531 scope.go:122] "RemoveContainer" containerID="92bc7b29e8281de1e85ad60206da4e2a6360b3b8d3f9f01079d8079c65bf564d" Oct 30 19:47:03.569117 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:03.569090 4531 scope.go:122] "RemoveContainer" containerID="e3a091e167ed169c13ade784e43a33b4ba4d7a246fd35d0035f29e87e0405650" Oct 30 19:47:03.577750 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:03.577731 4531 scope.go:122] "RemoveContainer" containerID="1e5140215ae2f72a7c94a2d576697ccd961bf53878035cf9e4139404a16b955f" Oct 30 19:47:04.048938 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:04.048905 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-frkss_429722d3-91b1-4c92-b78e-19a12bc5bac9/global-pull-secret-syncer/0.log" Oct 30 19:47:08.034717 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:08.034671 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-56jrt_a47b1484-d120-4c8e-ad86-1309eb472e84/manager/0.log" Oct 30 19:47:08.296664 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:08.296572 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sd88x_002053ca-649b-4a56-9ffd-25406d82fe99/konnectivity-agent/0.log" Oct 30 19:47:08.668607 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:08.668512 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-56.us-west-1.compute.internal_d156aeb8e5a3750f522b65c4725c3f82/haproxy/0.log" Oct 30 19:47:09.399694 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:09.399658 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-56jrt_a47b1484-d120-4c8e-ad86-1309eb472e84/manager/0.log" Oct 30 19:47:14.353301 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:14.353260 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-56jrt_a47b1484-d120-4c8e-ad86-1309eb472e84/manager/0.log" Oct 30 19:47:15.395361 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.395334 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-rkjnb_0c895121-1801-4d2a-a4d2-c33e167ec4b5/kuadrant-console-plugin/0.log" Oct 30 19:47:15.474389 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.474358 4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-v8nrk/perf-node-gather-daemonset-dcn7c"] Oct 30 19:47:15.474685 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.474673 4531 cpu_manager.go:414] "containerMap: removing container" podUID="0508b0cf-7cda-4723-b519-f84c16a29867" containerName="cleanup" Oct 30 19:47:15.474731 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.474686 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0508b0cf-7cda-4723-b519-f84c16a29867" containerName="cleanup" Oct 30 19:47:15.474731 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.474708 4531 cpu_manager.go:414] "containerMap: removing container" podUID="efe5fedb-d8ee-418b-89d6-1d1891e436df" containerName="istio-proxy" Oct 30 19:47:15.474731 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.474714 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efe5fedb-d8ee-418b-89d6-1d1891e436df" containerName="istio-proxy" Oct 30 19:47:15.474731 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.474729 4531 cpu_manager.go:414] "containerMap: removing container" podUID="97604a9b-3f63-49d0-8377-4d7a30fa4a08" containerName="revoke-keys" Oct 30 19:47:15.474865 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.474734 4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97604a9b-3f63-49d0-8377-4d7a30fa4a08" containerName="revoke-keys" Oct 30 19:47:15.474865 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.474786 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="efe5fedb-d8ee-418b-89d6-1d1891e436df" containerName="istio-proxy" Oct 30 19:47:15.474865 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.474795 4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="97604a9b-3f63-49d0-8377-4d7a30fa4a08" containerName="revoke-keys" Oct 30 19:47:15.477947 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.477927 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v8nrk/perf-node-gather-daemonset-dcn7c" Oct 30 19:47:15.532935 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.532911 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v8nrk\"/\"kube-root-ca.crt\"" Oct 30 19:47:15.535265 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.535250 4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v8nrk\"/\"openshift-service-ca.crt\"" Oct 30 19:47:15.582585 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.582549 4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-v8nrk\"/\"default-dockercfg-6bfd2\"" Oct 30 19:47:15.584929 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.584900 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v8nrk/perf-node-gather-daemonset-dcn7c"] Oct 30 19:47:15.590465 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.590439 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b19ff2aa-ec57-4dcb-915e-24bbf877b185-proc\") pod \"perf-node-gather-daemonset-dcn7c\" (UID: \"b19ff2aa-ec57-4dcb-915e-24bbf877b185\") " pod="openshift-must-gather-v8nrk/perf-node-gather-daemonset-dcn7c" Oct 30 19:47:15.590609 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.590478 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b19ff2aa-ec57-4dcb-915e-24bbf877b185-sys\") pod \"perf-node-gather-daemonset-dcn7c\" (UID: \"b19ff2aa-ec57-4dcb-915e-24bbf877b185\") " pod="openshift-must-gather-v8nrk/perf-node-gather-daemonset-dcn7c" Oct 30 19:47:15.590609 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.590504 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b19ff2aa-ec57-4dcb-915e-24bbf877b185-podres\") pod \"perf-node-gather-daemonset-dcn7c\" (UID: \"b19ff2aa-ec57-4dcb-915e-24bbf877b185\") " pod="openshift-must-gather-v8nrk/perf-node-gather-daemonset-dcn7c" Oct 30 19:47:15.590757 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.590686 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b19ff2aa-ec57-4dcb-915e-24bbf877b185-lib-modules\") pod \"perf-node-gather-daemonset-dcn7c\" (UID: \"b19ff2aa-ec57-4dcb-915e-24bbf877b185\") " pod="openshift-must-gather-v8nrk/perf-node-gather-daemonset-dcn7c" Oct 30 19:47:15.590833 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.590808 4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7t5j\" (UniqueName: \"kubernetes.io/projected/b19ff2aa-ec57-4dcb-915e-24bbf877b185-kube-api-access-z7t5j\") pod \"perf-node-gather-daemonset-dcn7c\" (UID: \"b19ff2aa-ec57-4dcb-915e-24bbf877b185\") " pod="openshift-must-gather-v8nrk/perf-node-gather-daemonset-dcn7c" Oct 30 19:47:15.691499 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.691397 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b19ff2aa-ec57-4dcb-915e-24bbf877b185-lib-modules\") pod \"perf-node-gather-daemonset-dcn7c\" (UID: \"b19ff2aa-ec57-4dcb-915e-24bbf877b185\") " pod="openshift-must-gather-v8nrk/perf-node-gather-daemonset-dcn7c" Oct 30 19:47:15.691499 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.691453 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7t5j\" (UniqueName: \"kubernetes.io/projected/b19ff2aa-ec57-4dcb-915e-24bbf877b185-kube-api-access-z7t5j\") pod \"perf-node-gather-daemonset-dcn7c\" (UID: \"b19ff2aa-ec57-4dcb-915e-24bbf877b185\") " pod="openshift-must-gather-v8nrk/perf-node-gather-daemonset-dcn7c" Oct 30 19:47:15.691499 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.691485 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b19ff2aa-ec57-4dcb-915e-24bbf877b185-proc\") pod \"perf-node-gather-daemonset-dcn7c\" (UID: \"b19ff2aa-ec57-4dcb-915e-24bbf877b185\") " pod="openshift-must-gather-v8nrk/perf-node-gather-daemonset-dcn7c" Oct 30 19:47:15.691797 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.691508 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b19ff2aa-ec57-4dcb-915e-24bbf877b185-sys\") pod \"perf-node-gather-daemonset-dcn7c\" (UID: \"b19ff2aa-ec57-4dcb-915e-24bbf877b185\") " pod="openshift-must-gather-v8nrk/perf-node-gather-daemonset-dcn7c" Oct 30 19:47:15.691797 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.691525 4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b19ff2aa-ec57-4dcb-915e-24bbf877b185-podres\") pod \"perf-node-gather-daemonset-dcn7c\" (UID: \"b19ff2aa-ec57-4dcb-915e-24bbf877b185\") " pod="openshift-must-gather-v8nrk/perf-node-gather-daemonset-dcn7c" Oct 30 19:47:15.691797 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.691561 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b19ff2aa-ec57-4dcb-915e-24bbf877b185-lib-modules\") pod \"perf-node-gather-daemonset-dcn7c\" (UID: \"b19ff2aa-ec57-4dcb-915e-24bbf877b185\") " pod="openshift-must-gather-v8nrk/perf-node-gather-daemonset-dcn7c" Oct 30 19:47:15.691797 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.691659 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b19ff2aa-ec57-4dcb-915e-24bbf877b185-sys\") pod \"perf-node-gather-daemonset-dcn7c\" (UID: \"b19ff2aa-ec57-4dcb-915e-24bbf877b185\") " pod="openshift-must-gather-v8nrk/perf-node-gather-daemonset-dcn7c" Oct 30 19:47:15.691797 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.691660 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b19ff2aa-ec57-4dcb-915e-24bbf877b185-podres\") pod \"perf-node-gather-daemonset-dcn7c\" (UID: \"b19ff2aa-ec57-4dcb-915e-24bbf877b185\") " pod="openshift-must-gather-v8nrk/perf-node-gather-daemonset-dcn7c" Oct 30 19:47:15.691797 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.691697 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b19ff2aa-ec57-4dcb-915e-24bbf877b185-proc\") pod \"perf-node-gather-daemonset-dcn7c\" (UID: \"b19ff2aa-ec57-4dcb-915e-24bbf877b185\") " pod="openshift-must-gather-v8nrk/perf-node-gather-daemonset-dcn7c" Oct 30 19:47:15.754062 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.754030 4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7t5j\" (UniqueName: \"kubernetes.io/projected/b19ff2aa-ec57-4dcb-915e-24bbf877b185-kube-api-access-z7t5j\") pod \"perf-node-gather-daemonset-dcn7c\" (UID: \"b19ff2aa-ec57-4dcb-915e-24bbf877b185\") " pod="openshift-must-gather-v8nrk/perf-node-gather-daemonset-dcn7c" Oct 30 19:47:15.792403 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:15.792365 4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v8nrk/perf-node-gather-daemonset-dcn7c" Oct 30 19:47:16.087629 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:16.087594 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-56jrt_a47b1484-d120-4c8e-ad86-1309eb472e84/manager/0.log" Oct 30 19:47:16.186230 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:16.186203 4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v8nrk/perf-node-gather-daemonset-dcn7c"] Oct 30 19:47:16.188115 ip-10-0-133-56 kubenswrapper[4531]: W1030 19:47:16.188087 4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb19ff2aa_ec57_4dcb_915e_24bbf877b185.slice/crio-904fcd0b5a65ea8ccc7309296d272b3ea94653419271f609425edfa46637c8fd WatchSource:0}: Error finding container 904fcd0b5a65ea8ccc7309296d272b3ea94653419271f609425edfa46637c8fd: Status 404 returned error can't find the container with id 904fcd0b5a65ea8ccc7309296d272b3ea94653419271f609425edfa46637c8fd Oct 30 19:47:16.346808 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:16.346724 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v8nrk/perf-node-gather-daemonset-dcn7c" event={"ID":"b19ff2aa-ec57-4dcb-915e-24bbf877b185","Type":"ContainerStarted","Data":"bdc57c4d4846d7b507e1cc99f100eabaccde0c05b86d6efc8f202444f171d45e"} Oct 30 19:47:16.346808 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:16.346787 4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v8nrk/perf-node-gather-daemonset-dcn7c" event={"ID":"b19ff2aa-ec57-4dcb-915e-24bbf877b185","Type":"ContainerStarted","Data":"904fcd0b5a65ea8ccc7309296d272b3ea94653419271f609425edfa46637c8fd"} Oct 30 19:47:16.346997 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:16.346840 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-v8nrk/perf-node-gather-daemonset-dcn7c" Oct 30 19:47:20.904190 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:20.904153 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-56jrt_a47b1484-d120-4c8e-ad86-1309eb472e84/manager/0.log" Oct 30 19:47:22.277782 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:22.277746 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-56jrt_a47b1484-d120-4c8e-ad86-1309eb472e84/manager/0.log" Oct 30 19:47:22.363619 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:22.363584 4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-v8nrk/perf-node-gather-daemonset-dcn7c" Oct 30 19:47:22.479625 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:22.479571 4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-v8nrk/perf-node-gather-daemonset-dcn7c" podStartSLOduration=7.479556626 podStartE2EDuration="7.479556626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:16.463578012 +0000 UTC m=+4396.490682970" watchObservedRunningTime="2026-10-30 19:47:22.479556626 +0000 UTC m=+4402.506661586" Oct 30 19:47:26.397885 ip-10-0-133-56 kubenswrapper[4531]: E1030 19:47:26.397846 4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v9jf7" containerName="extractor" Oct 30 19:47:27.033381 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:27.033338 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tvdr4_d36902b3-f9a4-4a16-b171-90e6eef10269/node-exporter/0.log" Oct 30 19:47:27.446699 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:27.446603 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tvdr4_d36902b3-f9a4-4a16-b171-90e6eef10269/kube-rbac-proxy/0.log" Oct 30 19:47:27.860189 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:27.860159 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tvdr4_d36902b3-f9a4-4a16-b171-90e6eef10269/init-textfile/0.log" Oct 30 19:47:28.039538 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:28.039504 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9468cdd5d-vhp9x_2650a306-fa87-40d3-b893-5480a959ac51/kube-auth-proxy/0.log" Oct 30 19:47:28.481265 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:28.481113 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-6khdd_f67d6e91-06ca-4538-8c0e-40d5421544b6/istio-proxy/0.log" Oct 30 19:47:29.468403 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:29.468373 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9468cdd5d-vhp9x_2650a306-fa87-40d3-b893-5480a959ac51/kube-auth-proxy/0.log" Oct 30 19:47:29.896733 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:29.896701 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-66db998887-6khdd_f67d6e91-06ca-4538-8c0e-40d5421544b6/istio-proxy/0.log" Oct 30 19:47:30.806578 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:30.806544 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bc2edd37-18e4-4982-8758-c45199e53612/prometheus/0.log" Oct 30 19:47:31.223874 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:31.223781 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bc2edd37-18e4-4982-8758-c45199e53612/config-reloader/0.log" Oct 30 19:47:31.642244 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:31.642212 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bc2edd37-18e4-4982-8758-c45199e53612/thanos-sidecar/0.log" Oct 30 19:47:32.054819 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:32.054785 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea583398-jdbxh"] Oct 30 19:47:32.060765 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:32.060736 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bc2edd37-18e4-4982-8758-c45199e53612/kube-rbac-proxy-web/0.log" Oct 30 19:47:32.103318 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:32.103277 4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c907a8b-5wvmk"] Oct 30 19:47:32.104758 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:32.104732 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea583398-jdbxh"] Oct 30 19:47:32.110155 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:32.107672 4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c907a8b-5wvmk"] Oct 30 19:47:32.476104 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:32.476020 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bc2edd37-18e4-4982-8758-c45199e53612/kube-rbac-proxy/0.log" Oct 30 19:47:32.890386 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:32.890352 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bc2edd37-18e4-4982-8758-c45199e53612/kube-rbac-proxy-thanos/0.log" Oct 30 19:47:33.338468 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:33.338437 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bc2edd37-18e4-4982-8758-c45199e53612/init-config-reloader/0.log" Oct 30 19:47:34.098470 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:34.098438 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71a76f93-3d23-43e6-a3ac-c63938d30f53" path="/var/lib/kubelet/pods/71a76f93-3d23-43e6-a3ac-c63938d30f53/volumes" Oct 30 19:47:34.098918 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:34.098876 4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfe5743d-6d47-48a3-b86c-927895ba3caf" path="/var/lib/kubelet/pods/cfe5743d-6d47-48a3-b86c-927895ba3caf/volumes" Oct 30 19:47:34.592664 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:34.592635 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-lwqfl_620dbcf1-8778-481c-b395-82a607eab38f/prometheus-operator-admission-webhook/0.log" Oct 30 19:47:35.016431 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:35.016403 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5795d67875-qlc84_969e9a2a-59b9-48fd-978a-9027833aeb8d/telemeter-client/0.log" Oct 30 19:47:35.315577 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:35.315493 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-rkjnb_0c895121-1801-4d2a-a4d2-c33e167ec4b5/kuadrant-console-plugin/0.log" Oct 30 19:47:35.493282 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:35.493248 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5795d67875-qlc84_969e9a2a-59b9-48fd-978a-9027833aeb8d/reload/0.log" Oct 30 19:47:35.908180 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:35.908148 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5795d67875-qlc84_969e9a2a-59b9-48fd-978a-9027833aeb8d/kube-rbac-proxy/0.log" Oct 30 19:47:37.003089 ip-10-0-133-56 kubenswrapper[4531]: I1030 19:47:37.003063 4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-rkjnb_0c895121-1801-4d2a-a4d2-c33e167ec4b5/kuadrant-console-plugin/0.log"